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Preface 


In  early  2008,  the  Office  of  the  Deputy  Assistant  Secretary  of  Defense  for  Partnership  Strategy 
and  Stability  Operations  asked  the  RAND  Corporation  to  develop  a  handbook  to  support  the 
monitoring  and  evaluation  (M&E)  of  humanitarian  assistance  (HA)  projects  funded  by  Over¬ 
seas  Humanitarian,  Disaster,  and  Civic  Aid  (OHDACA)  appropriations.  This  report  describes 
the  process  undertaken  by  the  RAND  research  team  and  contains  a  prototype  version  of  the 
handbook.  This  report  and,  in  particular,  the  prototype  handbook  should  be  of  interest  to 
U.S.  Department  of  Defense  (DoD)  staff  responsible  for  HA  project  planning,  execution,  and 
monitoring  and  those  in  the  military  chain  of  command  who  are  interested  in  higher-order 
program  or  strategic  evaluations  that  draw  from  project  assessments.  It  should  also  be  of  inter¬ 
est  to  civilian  agencies  that  interact  with  DoD  counterparts  in  the  planning  or  follow-up  to 
DoD  HA  projects  in  the  field  and  to  members  of  Congress  and  their  staffs,  as  evolving  policies 
and  requirements  related  to  M&E  in  DoD  derive  in  large  part  from  statutory  requirements. 

This  research  was  sponsored  by  the  Office  of  the  Deputy  Assistant  Secretary  of  Defense 
for  Partnership  Strategy  and  Stability  Operations  and  conducted  jointly  by  RAND  Health’s 
Center  for  Military  Health  Policy  Research  and  the  International  Security  and  Defense  Policy 
Center  of  the  RAND  National  Defense  Research  Institute  (NDRI).  The  Center  for  Military 
Health  Policy  Research  taps  RAND  expertise  in  both  defense  and  health  policy  to  conduct 
research  for  the  Department  of  Defense,  the  Veterans  Administration,  and  nonprofit  organiza¬ 
tions.  RAND  Health  aims  to  transform  the  well-being  of  all  people  by  solving  complex  prob¬ 
lems  in  health  and  health  care.  NDRI  is  a  federally  funded  research  and  development  center 
sponsored  by  the  Office  of  the  Secretary  of  Defense,  the  Joint  Staff,  the  Unified  Combatant 
Commands,  the  Navy,  the  Marine  Corps,  the  defense  agencies,  and  the  defense  Intelligence 
Community. 

For  more  information  on  the  Center  for  Military  Health  Policy  Research,  see  http:// 
www.rand.org/multi/military/  or  contact  the  co-directors  (contact  information  is  provided  on 
the  web  page).  For  more  information  on  the  International  Security  and  Defense  Policy  Center, 
see  http://www.rand.org/nsrd/about/isdp.html  or  contact  the  director  (contact  information  is 
provided  on  the  web  page). 
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Summary 


The  impetus  for  the  development  of  a  handbook  to  guide  the  assessment  of  military  humani¬ 
tarian  assistance  (HA)  projects  came  from  a  January  2008  workshop  with  combatant  com¬ 
mand  (COCOM)  HA  managers,  “Monitoring  and  Evaluation  of  DoD  Humanitarian  Assis¬ 
tance  Programs.”  Participants  felt  that  a  user-friendly  handbook  to  clarify  overall  monitoring 
and  evaluation  (M&E)  concepts  and  to  facilitate  project  assessment  would  be  particularly 
valuable  to  those  responsible  for  overseeing  the  execution  of  HA  projects  in  the  field.  Recent 
policy  guidance  and  the  development  of  this  handbook  are  part  of  an  increased  emphasis  on 
assessment  across  the  U.S.  Department  of  Defense  (DoD). 


Developing  the  Handbook 

The  project  team  took  a  broad  and  systematic  approach  to  developing  and  refining  the  proto¬ 
type  handbook,  which  is  included  as  an  appendix  to  this  report.  To  develop  the  initial  draft  of 
the  handbook,  we  first  reviewed  published  reports  from  various  government,  nongovernment, 
and  academic  sources  to  better  understand  different  approaches  to  project  assessment.  Second, 
we  searched  the  Overseas  Humanitarian  Assistance  Shared  Information  System  (OHASIS) 
database  to  gain  a  better  understanding  of  existing  and  proposed  Overseas  Humanitarian, 
Disaster,  and  Civic  Aid  (OHDACA)  HA  projects.  Finally,  we  conducted  semistructured  inter¬ 
views,  first  with  COCOM  HA  managers  and  then  with  others,  to  elicit  views  on  perceived 
opportunities  and  concerns  related  to  the  assessment  of  HA  projects. 

Our  interviews  suggested  that  the  handbook  should  include  both  an  educational 
component — a  primer  on  M&E — and  a  step-by-step  user’s  guide  for  conducting  project 
assessment  activities.  The  primer  was  developed  to  provide  those  tasked  with  project  assess¬ 
ment  with  a  concise,  yet  comprehensive,  introduction  to  M&E  terminology,  concepts,  and 
approaches,  with  an  emphasis  on  the  ability  to  coordinate  and  communicate  with  civilian  agen¬ 
cies.  The  user’s  guide  was  developed  to  walk  users  through  the  concrete  steps  of  project-level 
assessment — from  HA  project  conception  and  planning  to  project  execution  and  monitoring 
for  progress  and  results. 

After  we  completed  the  first  full  draft  of  the  handbook  in  December  2008,  we  asked 
stakeholders  in  DoD  as  well  as  experts  from  the  U.S.  Agency  for  International  Development 
(USAID)  and  the  U.S.  Department  of  State’s  Bureau  of  Population,  Refugees,  and  Migration 
to  review  the  draft  and  provide  written  feedback.  As  another  avenue  of  feedback  intended  to 
inform  revision  of  the  first  draft  handbook,  we  sought  at  least  one  or  two  countries  where  it 
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could  be  field-tested.  Ultimately,  DoD  HA  staff  in  Albania  and  Morocco  agreed  to  carry  out 
the  pilot  tests. 

Overall,  stakeholders  who  reviewed  the  handbook  felt  that  it  represented  a  strong  contri¬ 
bution  to  the  M&E  literature,  not  only  for  DoD  but  also  for  the  larger  international  develop¬ 
ment  community.  These  reviewers  provided  important  feedback  that  informed  our  subsequent 
revisions.  Those  who  pilot-tested  the  handbook  in  Albania  and  Morocco  felt  that  it  was  a 
valuable  resource  aimed  at  the  appropriate  level  for  those  conducting  field-level  HA  project 
assessments.  They  provided  useful  comments  regarding  ways  to  streamline  and  simplify  the 
document. 

We  revised  the  handbook  based  on  the  constructive  feedback  provided  through  the  review 
and  pilot-testing  processes. 


Discussion  and  Recommendations 

Although  we  received  extensive  and  valuable  feedback  from  reviews  of  the  initial  draft,  pilot¬ 
testing  was  limited  to  two  countries,  a  small  number  of  projects,  and  a  relatively  short  time 
frame.  Thus,  several  aspects  of  the  handbook’s  use  could  not  be  tested,  including  its  use  across 
all  types  of  HA  projects,  its  use  over  time  (particularly  when  a  project  is  handed  off  from  one 
team  to  another),  and  the  feasibility  and  usefulness  of  compiling  data  from  many  individual 
projects  to  inform  higher-level  assessments.  Because  of  the  limited  pilot  testing  to  date,  we 
recommend  that  the  prototype  handbook  undergo  further  testing,  if  possible,  before  it 
is  broadly  used. 

Most  staff  responsible  for  HA  projects  are  currently  not  undertaking  any  project  assess¬ 
ment  activities,  including  fulfilling  the  existing  requirement  to  complete  an  after- action  report, 
so  the  perception  is  that  any  project  assessment  activity  will  add  burden.  It  is  hoped,  however, 
that  the  assessment  approach  outlined  in  the  prototype  handbook  will  actually  reduce  the 
burden  on  staff  after  an  initial  period  of  uptake  and  adjustment.  However,  the  potential  for 
reduced  burden — both  real  and  perceived — will  be  contingent  on  DoD  both  formally  requir¬ 
ing  HA  project  assessment  and  enforcing  that  requirement.  DoD  should  explicitly  require 
HA  project  assessment  and  enforce  this  requirement  (e.g.,  new  project  nominations  should 
not  be  accepted  unless  assessment  requirements  from  previous  projects  are  met). 

The  level  of  burden  related  to  conducting  project  assessment  activities  will  also  depend 
on  the  way  in  which  data  are  captured  and  stored.  To  reduce  the  burden  of  HA  project-level 
assessment  and  to  allow  generated  data  to  be  compiled  to  inform  higher-level  assessments, 
the  collection  of  project  assessment  information  as  outlined  in  the  prototype  handbook 
should  be  incorporated  within  current  and  future  efforts  to  upgrade  and  improve  the 
OHASIS  database. 

Without  information  captured  in  OHASIS  or  another  similar  system,  project-level  assess¬ 
ments  will  be  of  limited  use.  DoD  will  need  to  decide  whether  it  will  begin  to  require  project- 
level  assessment  using  the  handbook  before  this  retrofitting  of  OHASIS  is  complete  or  whether 
it  would  prefer  to  wait  until  the  system  is  redesigned  to  support  assessment  activities.  While 
starting  sooner  may  be  beneficial  from  the  perspective  of  training  and  instilling  best  practices 
in  project  management,  the  potential  for  perceived  additional  burden  and  limited  usefulness  is 
higher  without  the  systems  support  that  would  be  provided  by  a  redesigned  OHASIS  database. 
To  the  extent  that  funds  are  available,  the  capture  of  HA  project  assessment  information 
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as  outlined  in  the  handbook  should  be  incorporated  into  OHASIS  before  HA  project 
assessment  is  required. 

Although  there  are  several  different  handbook  formats  that  could  be  tested  in  an 
additional  round  of  pilot  tests,  based  on  the  feedback  that  we  received,  we  recommend  a 
letter-sized,  spiral-bound  handbook  with  a  flexible  protective  cover  and  a  CD  insert  that 
includes  an  electronic  version  of  the  handbook  and  all  handbook  worksheets. 

It  is  hoped  that  a  final  handbook,  though  designed  with  a  specific  focus  in  mind,  will  be 
used  more  broadly  over  time  by  DoD  and  potentially  beyond.  To  support  its  sustainable  use 
by  DoD,  in  particular,  the  handbook  should  be  incorporated  into  the  formal  DoD  hand¬ 
book  series. 


Prototype  Handbook 

The  prototype  handbook  is  presented  as  an  appendix  to  this  report.  It  includes  three  major 
parts:  an  introduction,  an  M&E  primer,  and  a  step-by-step  user’s  guide  for  project  assessment. 
The  M&E  primer  educates  handbook  users  on  basic  M&E  concepts  and  terms  (including 
those  that  go  beyond  the  scope  of  project-level  assessment),  how  to  plan  for  project  assessment, 
and  the  appropriate  use  of  indicators  in  various  types  of  project  assessment.  The  user’s  guide 
presents  11  project  assessment  steps  and  includes  various  worksheets  to  facilitate  assessment 
planning  and  data  collection.  In  addition,  the  prototype  handbook  includes  its  own  table 
of  contents  and  appendixes  with  supplemental  information  on  community  consultation  and 
avoiding  bias  in  project  assessment,  as  well  as  a  glossary  of  M&E  terms. 
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Introduction 


Background 

The  impetus  for  the  development  of  a  handbook  to  guide  the  assessment  of  military  humanitar¬ 
ian  assistance  (HA)  projects  came  from  a  January  2008  workshop  with  combatant  command 
(COCOM)  HA  managers,  “Monitoring  and  Evaluation  of  DoD  Humanitarian  Assistance 
Programs.”  The  workshop  provided  a  forum  in  which  to  discuss  and  better  understand  basic 
monitoring  and  evaluation  (M&E)  concepts,  recent  policy  guidance  requiring  the  measure¬ 
ment  of  HA  project  effectiveness,  and  future  expectations  with  regard  to  project  assessment. 
Participants  felt  that  a  user-friendly  handbook  to  clarify  M&E  concepts  and  to  facilitate  proj¬ 
ect  assessment  would  be  particularly  valuable  to  those  responsible  for  overseeing  the  execution 
of  HA  projects  in  the  field. 

To  help  ensure  that  the  goals  and  objectives  of  its  HA  missions  are  met,  the  U.S.  Depart¬ 
ment  of  Defense  (DoD)  has  included  M&E  requirements  and  resources  in  its  Policy  Guidance 
for  Overseas  Humanitarian  Assistance.  The  fiscal  year  (FY)  2008  guidance  included  require¬ 
ments  for  ensuring  that  projects  are  sustainable,  for  coordinating  with  other  relevant  agencies 
and  stakeholders,  and  for  measuring  project  effectiveness.1  The  FY  2010  guidance  explicitly 
noted  that  “metrics  are  essential  for  measuring  achievement,”  “project  effectiveness  should  be 
considered,”  and  the  “development  and  application  of  specific  HA  metrics  is  pending  estab¬ 
lishment  of  a  funding  source  for  the  collection  of  metrics,”  with  further  guidance  to  follow.2 

The  FY  2008  guidance  notes  that  the  Overseas  Humanitarian,  Disaster,  and  Civic  Aid 
(OHDACA)  program  falls  under  the  joint  oversight  of  the  Deputy  Assistant  Secretary  of 
Defense  for  Partnership  Strategy  and  Stability  Operations  and  the  Defense  Security  Coopera¬ 
tion  Agency  (DSCA),  and,  as  such,  should  strive  to  achieve  such  security  cooperation  objec¬ 
tives  as  “improving  DoD  visibility,  access,  and  influence  in  a  partner  nation  or  region,  gener¬ 
ating  long-term  public  relations  and  goodwill  for  DoD,  and  promoting  interoperability  and 
coalition-building  with  foreign  military  and  civilian  counterparts.”3  The  guidance  further 
emphasizes  capacity-building  (i.e.,  the  transfer  of  knowledge  and  skills  to  the  host  nation), 
sustainability,  and  interagency  coordination  as  vital  to  HA  projects. 

DoD  policy  guidance  and  the  development  of  this  handbook  are  part  of  an  increased 
emphasis  on  assessment  across  the  department.  DoD  is  placing  greater  emphasis  on  assessing 


1  U.S.  Department  of  Defense,  Policy  Guidance  for  FY08  Overseas  Humanitarian  Assistance ,  September  19,  2007. 

2  U.S.  Department  of  Defense,  Policy  Guidance  for  DoD  Overseas  Humanitarian  Assistance  Program  (HAP)  FY  2010, 
Washington,  D.C.,  November  18,  2009. 

3  U.S.  Department  of  Defense,  2007. 


1 


2  Developing  a  Prototype  Handbook  for  Monitoring  and  Evaluating  DoD  Humanitarian  Assistance  Projects 


progress  toward  strategic  goals  to  better  inform  future  priorities,  planning,  requirements,  and 
resource  allocation.  For  example,  the  Guidance  for  the  Employment  of  the  Force  (GEF)  is  a 
guidance  document  from  the  Office  of  the  Secretary  of  Defense  (OSD)  that  defines  functional 
and  theater  strategic  “end  states”  and  requires  an  assessment  of  progress  toward  achieving  these 
end  states.  Relevant  to  HA  project  assessment,  the  GEF  focuses  on  building  partner  capacity 
in  the  context  of  security  cooperation  programming  and  calls  for  assessments  to  build  up  from 
local-level  (project)  outputs  to  strategic-level  goals. 


Project  Goal  and  Objective 

The  goal  of  this  project  was  to  develop  an  educational  and  practical  tool  to  support  an  expected 
requirement  that  OHDACA-funded  HA  projects  be  monitored  and  evaluated  and  to  support 
the  larger  push  for  assessment  DoD-wide.  The  primer  was  developed  to  educate  novice  users  on 
basic  M&E  concepts  and  terms,  including  topic  areas  that  go  beyond  basic  project-level  assess¬ 
ment  (e.g.,  evaluating  the  effects  of  a  collection  of  projects,  or  program  evaluation).  Knowledge 
obtained  from  the  primer  should  assist  users  in  planning  for  and  conducting  project  assess¬ 
ments;  it  will  also  provide  a  broader  understanding  of  M&E  concepts,  terms,  and  methods, 
which  should  foster  communication  and  collaboration  with  civilian  counterparts. 

The  user’s  guide  is  aimed  at  “walking”  users,  with  knowledge  from  the  primer  in  hand, 
through  each  of  the  steps  required  for  project  assessment  (defined  here  as  monitoring  project 
inputs,  processes,  outputs,  and  outcomes,  but  not  necessarily  evaluating  why  or  how  results 
occur).  The  objective  was  to  develop  a  guide  and  associated  tools  to  generate  information  that 
would  be  useful  to  both  the  specific  project  team  and  those  higher  in  the  chain  of  command. 
That  is,  application  of  the  user’s  guide  should  produce  information  that  is  useful  for  assessing 
project-level  performance  and  effectiveness  as  well  as  information  that  can  inform  higher- 
level  program  assessments  aimed  at  evaluating  progress  toward  achieving  higher-order  DoD 
objectives. 

The  prototype  handbook,  included  as  an  appendix  to  this  report,  incorporates  feedback 
from  key  stakeholders  and  two  pilot  tests  of  an  earlier  draft  in  the  field.  Ideally,  it  should  be 
more  rigorously  field-tested  and  revised  again  based  on  those  tests  before  it  is  used  operation¬ 
ally  by  DoD.  The  handbook  is  intended  for  use  by  those  responsible  for  planning  and  execut¬ 
ing  HA  projects  and  generating  project-level  assessment  data,  especially  DoD  country-level 
staff  and  COCOM  HA  managers.  These  users  (likely,  civil  affairs  personnel)  might  already  be 
overburdened  and,  while  involved  in  development  activities,  may  not  be  M&E  experts.  The 
information  generated  through  the  use  of  the  handbook  will  also  be  of  interest  to  others  in  the 
DoD  chain  of  command,  particularly  those  with  an  interest  in  project-level  assessment  data 
as  they  relate  to  higher-order  (strategic)  objectives — for  example,  the  COCOM  commander, 
DSCA,  and  others. 

Although  the  prototype  handbook  included  here  focuses  on  assessment  of  OHDACA- 
funded  HA  projects  and  is  intended  primarily  for  use  by  those  responsible  for  executing 
OHDACA  projects,  a  final  handbook  may  be  useful  for  similar  or  related  DoD  efforts  through 
the  Humanitarian  Civic  Assistance,  Denton  and  Funded  Transportation,  Foreign  Disaster 
Relief,  and  Humanitarian  Assistance  Excess  Property  programs  and,  potentially,  to  users 
beyond  DoD,  such  as  those  working  on  projects  supported  by  the  U.S.  Agency  for  Interna¬ 
tional  Development  (USAID). 
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Organization  of  This  Report 

The  remainder  of  this  report  provides  information  on  the  approach  used  for  developing  and 
refining  the  prototype  handbook,  the  results  from  each  phase  of  our  work,  and  a  brief  discus¬ 
sion  of  remaining  issues  and  recommendations  for  finalizing  and  ultimately  using  the  hand¬ 
book.  The  prototype  handbook  itself  is  included  as  an  appendix.  It  consists  of  three  major 
parts:  an  introduction,  an  M&E  primer,  and  a  step-by-step  user’s  guide  for  project  assessment. 


Developing  the  Handbook 


Study  Approach 

The  project  team  took  a  broad  and  systematic  approach  to  developing  and  refining  the  proto¬ 
type  handbook  included  in  this  report. 

To  develop  the  initial  draft  of  the  handbook,  we  first  reviewed  published  reports  from 
various  government,  nongovernment,  and  academic  sources  to  better  understand  different 
approaches  to  project  assessment.  Specifically,  we  reviewed  documents  published  by  DoD; 
USAID;  the  U.S.  Department  of  State’s  Bureau  of  Population,  Refugees,  and  Migration 
(PRM);  the  World  Bank;  the  United  Nations  Development  Programme;  the  Sphere  Project; 
and  the  USAID-funded  MEASURE  (Monitoring  and  Evaluation  to  Assess  and  Use  Results) 
program.  Important  sources  of  information  can  be  found  in  the  references  section  of  the 
prototype  handbook.  As  part  of  this  exercise,  we  conducted  a  comprehensive  review  of  assess¬ 
ment  terms,  methods,  and  data  sources  that  are  used  by  DoD  and  other  relevant  organizations. 
The  primary  goal  of  the  literature  review  was  to  compare  military  and  civilian  approaches  to 
project  assessment  and  to  determine  how  to  link  them.  A  key  task  was  to  compare  assessment 
terms  and  definitions  and  to  establish  consistent  terminology  for  the  handbook  to  facilitate 
DoD  coordination  and  communication  on  assessment  with  other  U.S.  government  agencies, 
while  at  the  same  time  retaining  language  embedded  in  DoD  policy  and  doctrine. 

Second,  we  searched  the  Overseas  Humanitarian  Assistance  Shared  Information  System 
(OHASIS)  database  to  gain  a  better  understanding  of  existing  and  proposed  OHDACA  HA 
projects,  including  project  descriptions,  budgets,  timelines,  locations,  and  primary  activities. 
The  analyses  provided  the  basis  for  a  simple  typology  of  HA  projects  for  use  in  the  handbook’s 
development  and  subsequent  pilot-testing  activities. 

Third,  we  conducted  semistructured  interviews,  first  with  COCOM  HA  managers 
and  then  with  others,  to  elicit  views  on  perceived  opportunities  and  concerns  related  to  the 
assessment  of  HA  projects.  We  conducted  approximately  15  interviews  with  key  staff  from 
OSD,  DSCA,  the  COCOMs  (U.S.  Africa  Command  [USAFRICOM],  U.S.  European  Com¬ 
mand  [USEUCOM],  U.S.  Pacific  Command  [USPACOM],  and  U.S.  Northern  Command 
[USNORTHCOM]),  and  country-level  DoD  offices  (in  Moldova,  the  Philippines,  Sene¬ 
gal,  and  Ukraine).  At  the  COCOMs,  we  interviewed  HA  managers  and  COCOM  assess¬ 
ment  leaders.  At  the  country  level,  we  interviewed  Office  of  Defense  Cooperation  chiefs  and 
local  team  members  who  could  potentially  pilot  the  handbook  on  the  ground.  The  goal  of 
these  interviews  was  to  determine  what  HA  activities  were  currently  being  conducted,  which 
assessment-related  skills  these  relevant  DoD  staff  might  already  have,  and  what  types  of  infor- 
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mation  would  be  most  applicable  for  those  using  the  handbook — both  at  the  project  level  and 
among  those  interested  in  compiling  project-level  data  to  inform  broader  evaluations. 

The  semistructured  interviews  consisted  of  overview  questions  about  HA  and  related 
activities,  including  sources  of  funding  and  timelines;  the  process  of  project  nomination, 
approval,  and  funding;  the  types  of  projects  currently  funded  and  those  that  HA  managers 
might  like  to  see  funded  in  the  future;  how  the  HA  project  cycle  compares  to  other  interna¬ 
tional  development  project  cycles;  parties  responsible  for  collecting,  analyzing,  and  using  HA 
assessment  information;  the  characteristics  of  those  in  the  field  who  will  be  designing  and 
implementing  project  assessment  plans,  including  their  skill  levels;  views  on  the  benefits  of 
assessment  information  for  current  and  future  projects;  how  the  process  of  M&E  data  collec¬ 
tion  and  information-sharing  might  be  made  easier  and  more  streamlined;  how  assessment 
information  is  actually  used  up  the  chain  of  command  and  by  whom;  and  information  about 
project  resources  and  capacities  for  project  assessment. 


Developing  the  First  Draft  of  the  Handbook 

Our  interviews  confirmed  that  the  handbook  should  include  both  an  educational 
component — a  primer  on  M&E — and  a  step-by-step  user’s  guide  for  conducting  narrower 
project  assessment  activities.  The  primer  was  developed  to  provide  those  tasked  with  project 
assessment  with  a  concise,  yet  comprehensive,  introduction  to  M&E  terminology,  concepts, 
and  approaches,  with  an  emphasis  on  the  ability  to  coordinate  and  communicate  with  civilian 
agencies.  While  the  prototype  handbook  is  based  on  DoD  M&E  or  assessment  terms,  we  paid 
special  attention  to  the  linking  of  military  and  civilian  terminology  in  the  primer  to  facilitate 
interagency  coordination  on  the  ground.  The  user’s  guide  was  developed  to  walk  users  through 
the  concrete  steps  of  project-level  assessment — from  HA  project  conception  and  planning  to 
project  execution  and  monitoring  for  progress  and  results.  In  principle,  the  user’s  guide  and 
the  primer  can  each  stand  alone.  Those  already  experienced  with  M&E  or  assessment  can  go 
directly  to  the  user’s  guide  and  begin  project  assessment  work,  while  those  who  need  an  intro¬ 
duction  or  a  more  detailed  orientation  can  consult  some  or  all  of  the  primer. 

We  developed  the  prototype  handbook  according  to  a  set  of  usability  criteria  established 
at  the  outset  of  the  project  and  confirmed  during  our  interviews  with  stakeholders.  The  hand¬ 
book  is  intended  to  be 

•  user-friendly — approachable  and  easy  to  use  from  the  end  user’s  standpoint 

•  useful — focused  on  the  most  practical  elements  of  M&E  that  will  be  most  useful  to  the 
end  user  as  well  as  to  stakeholders  up  the  command  chain 

•  focused  on  priorities — grounded  by  DoD’s  strategic  objectives  and  explicit  HA  program 
goals 

•  succinct — as  brief  and  to-the-point  as  possible,  without  losing  key  information 

•  organized — taking  into  account  the  structure  of  guidebooks  from  other  agencies  and 
organizations 

•  compatible  within  DoD — compatible  with  existing  DoD  M&E  or  assessment  doctrine, 
policy,  frameworks,  tools,  metrics,  and  definitions 


Developing  the  Handbook  7 


•  compatible  with  other  key  agencies — compatible  with  M&E  terminology,  methods,  and 
metrics  used  by  key  agencies,  such  as  USAID,  the  U.S.  Department  of  State,  and  large 
international  and  nongovernmental  organizations 

•  generalizable — applicable  to  non-OHDACA  and  non-DoD  projects  by  drawing  on  widely 
used  M&E  frameworks  from  civilian  programs,  such  as  the  Sphere  Project  and  USAID’s 
MEASURE  project 

•  attributable — performance  measures  based  on  frameworks  that  help  establish  causal 
pathways  and,  thus,  reflect  results  that  are  likely  attributable  to  project  activities. 


Review  and  Feedback  from  Key  Stakeholders 

After  we  completed  the  first  full  draft  of  the  handbook  in  December  2008,  we  asked  stake¬ 
holders  in  DoD  (including  several  who  were  interviewed  earlier)  and  experts  from  USAID  and 
PRM  to  review  the  draft  and  provide  written  feedback.  In  January  2009,  through  OSD,  we 
distributed  the  handbook  for  comment  to  a  selected  group  of  approximately  20  people,  includ¬ 
ing  DSCA  staff,  COCOM  HA  managers,  staff  from  the  Office  of  the  Assistant  Secretary 
of  Defense  for  Health  Affairs,  staff  from  USAID  (including  its  senior  development  advisers 
[SDAs]  at  the  COCOMs),  and  staff  from  PRM. 

Pilot-Testing  in  the  Field 

As  another  avenue  for  feedback  intended  to  inform  the  revision  of  the  first  draft  of  the  hand¬ 
book,  we  sought  at  least  one  or  two  countries  where  it  could  be  field-tested.  Ideally,  these  tests 
would  address  projects  that  fall  into  one  of  the  main  types  of  HA  projects  and  cover  different 
phases  of  the  project  cycle  (e.g.,  planning,  execution,  post-completion).  We  contacted  coun¬ 
try  team  members  in  11  countries  across  USAFRICOM  (Senegal,  Kenya,  Morocco,  Zimba¬ 
bwe,  Angola,  Botswana,  and  Liberia),  USEUCOM  (Moldova,  Albania,  and  Ukraine),  and 
USPACOM  (Philippines)  about  the  possibility  of  field-testing  the  handbook.  Ultimately,  staff 
in  Albania  and  Morocco  agreed  to  carry  out  the  pilot  tests.  To  assess  the  practical  utility  of  the 
handbook  based  on  just  a  small  number  of  field  tests,  we  aimed  to  gather  pilot-test  informa¬ 
tion  on  more  than  one  project  per  country — similar  projects  at  different  points  in  the  project 
cycle.  We  provided  staff  in  these  two  countries  with  materials  to  introduce  and  orient  them 
to  the  handbook,  conducted  a  “kick-off”  telephone  meeting,  and  made  ourselves  available  to 
the  field  teams  for  any  technical  or  logistical  questions  that  might  arise  as  they  applied  the 
handbook  to  planning  and  monitoring  projects  in  their  respective  countries.  We  asked  them  to 
complete  a  general  feedback  form  to  rate  and  provide  comments  on  the  content,  organization, 
format,  clarity,  and  expected  utility  of  the  handbook.  We  also  asked  them  to  submit  relevant 
worksheets  (i.e.,  the  “objective  tree”  template  or  other  relevant  indicator  worksheets)  from  the 
user’s  guide  that  they  used  to  monitor  their  projects  and  to  comment  on  the  usability  of  those 
particular  components.  The  process  of  seeking  sites  for  pilot-testing  began  in  March  2009,  and 
the  two  pilot  tests  were  completed  by  October  2009. 

Revising  the  Handbook 

We  revised  the  handbook  based  on  written  comments  from  key  stakeholders  as  well  as  written 
and  oral  comments  from  the  country  team  staff  leading  the  pilot  tests  in  Morocco  and  Alba¬ 
nia.  We  considered  all  suggestions  and  comments,  particularly  those  associated  with  improv- 
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ing  the  utility  of  the  handbook  in  the  held,  especially  if  they  came  from  those  who  pilot-tested 
it  in  the  held.  The  reviews  underwent  three  independent  readings  to  determine  key  action 
points.  Responses  to  each  action  point  were  developed  collaboratively  by  the  RAND  team.  The 
following  section  describes  the  hrst  draft  of  the  handbook,  key  comments  received  from  the 
reviews  and  pilot-testing,  and  the  major  changes  made  to  the  prototype  handbook. 


Results 

Eleven  representatives  from  DoD  (staff  from  OSD’s  Ofhce  of  Partnership  Strategy,  staff  from 
the  Ofhce  of  the  Assistant  Secretary  of  Defense  for  Health  Affairs,  and  COCOM  HA  manag¬ 
ers),  USAID,  and  PRM  provided  feedback  on  the  draft  handbook.  We  received  comments  from 
four  of  the  six  COCOM  HA  managers  and  two  of  the  hve  USAID  SDAs  at  the  COCOMs. 
Overall,  the  reviewers  felt  that  the  handbook  itself  represented  a  strong  contribution  to  the 
M&E  literature,  not  only  for  DoD  but  also  for  the  larger  international  development  commu¬ 
nity.  One  representative  from  USAID  found  the  handbook  to  be  “one  of  the  most  comprehen¬ 
sive  and  concise  overviews  of  M&E”  ever  seen,  further  stating  that  USAID  was  interested  in 
using  it  for  training  because  it  “is  clear  in  its  explanations,  builds  in  a  logical  order,  and  covers 
numerous  aspects  of  both  understanding  and  implementing  good  M&E.”  Reviewers  provided 
important  feedback  on  the  content  and  sequencing  of  the  handbook,  the  scenario  used  to  illus¬ 
trate  the  assessment  process,  and  the  potential  burden  associated  with  implementing  assess¬ 
ment  activities. 

Despite  the  notable  enthusiasm  expressed  by  most  country  team  representatives  contacted 
to  field-test  the  handbook,  we  received  feedback  from  only  two  of  the  1 1  country  teams  (Alba¬ 
nia  and  Morocco).  Each  field-tester  provided  ratings  and  written  comments  on  the  handbook, 
focusing  on  content  and  ease  of  use,  using  the  feedback  form  we  provided.  In  one  case,  we 
also  received  examples  of  completed  indicator  worksheets,  which  helped  us  understand  how 
they  could  be  improved  to  facilitate  their  use.  We  received  further  comments  during  post-pilot 
phone  calls,  which  also  contributed  to  handbook  revisions.  In  Albania,  the  local  HA  project 
manager  felt  that  the  handbook  was  a  valuable  resource  aimed  at  the  appropriate  level  for  those 
conducting  field-level  project  assessments.  In  particular,  he  felt  that  the  forms  and  checklists 
for  planning  and  data  collection  were  very  valuable,  as  was  the  ability  for  each  section  of  the 
handbook  (i.e.,  the  primer  and  user’s  guide)  to  stand  alone.  In  Morocco,  the  civil  affairs  officer 
who  field-tested  the  handbook  was  experienced  with  M&E  concepts  and  found  the  indicator 
worksheets  easy  to  use;  he  suggested  simplifications  to  make  them  clearer  and  less  repetitive. 
In  both  the  Morocco  and  Albania  pilot  tests,  handbook  users  considered  the  project  assessment 
as  a  task  in  addition  to  existing  HA  project  requirements — as  opposed  to  an  integral  part  of 
their  work. 

The  prototype  handbook  reflects  revisions  based  on  the  constructive  feedback  provided 
through  the  review  and  pilot-testing  processes  described  here. 


Discussion  and  Recommendations 

As  we  developed  the  handbook,  we  sought  to  ensure  that  it  was  user-friendly  and  useful  to 
those  who  would  be  required  to  implement  it,  focused  on  DoD  priorities,  succinct,  well  orga- 
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nized,  compatible  within  DoD,  compatible  with  key  civilian  agencies,  generalizable  beyond 
OHDACA  and  DoD,  and  helpful  in  linking  HA  projects  to  specific  project-  and  strategic-level 
outcomes.  Although  we  received  extensive  and  valuable  feedback  from  reviews  of  the  initial 
draft,  pilot-testing  was  limited  to  two  countries,  a  small  number  of  projects,  and  a  relatively 
short  time  frame.  Thus,  several  aspects  of  the  handbook’s  use  could  not  be  tested,  including 
its  use  across  all  types  of  HA  projects,  its  use  over  time  (particularly  when  a  project  is  handed 
off  from  one  team  to  another),  and  the  feasibility  and  usefulness  of  compiling  data  from  many 
individual  projects  to  inform  higher-level  assessments.  Because  of  the  limited  pilot  testing  to 
date,  we  recommend  that  the  prototype  handbook  undergo  further  testing,  if  possible, 
before  it  is  broadly  used.  This  would  allow  the  incorporation  of  additional  feedback  from  the 
field  to  ensure  that  the  usability  criteria  are  met  to  the  greatest  extent  possible. 

Although  additional  pilot-testing  of  the  handbook  could  help  ensure  that  the  handbook 
meets  all  of  usability  criteria,  there  are  two  related  issues  that  further  pilot-testing  will  not 
adequately  address  unless  specific  supportive  actions  are  taken  by  DoD:  the  issue  of  additional 
burden  and  the  issue  of  usefulness  for  compiling  project-level  assessment  data  to  inform  higher- 
level  assessments.  The  first  issue  was  questioned  by  several  reviewers  and  mentioned  as  a  con¬ 
cern  in  both  pilot  tests.  Most  staff  responsible  for  HA  projects  are  currently  not  undertaking 
any  project  assessment  activities,  including  fulfilling  the  existing  requirement  to  complete  an 
after-action  report,  so  the  perception  is  that  any  project  assessment  activity  will  add  burden. 
It  is  hoped,  however,  that  the  assessment  approach  outlined  in  the  prototype  handbook  will 
actually  reduce  the  burden  on  staff  after  an  initial  period  of  uptake  and  adjustment  because 
it  will  make  project-relevant  information  available  and  accessible  to  successive  teams  that  are 
responsible  for  HA  projects,  thus  reducing  the  burden  of  project  reporting  requirements.  How¬ 
ever,  the  potential  for  reduced  burden — both  real  and  perceived — will  be  contingent  on  DoD 
both  formally  requiring  HA  project  assessment  and  enforcing  that  requirement.  DoD  should 
explicitly  require  HA  project  assessment  and  enforce  this  requirement  (e.g.,  new  project 
nominations  should  not  be  accepted  unless  assessment  requirements  from  previous  projects  are 
met).  Without  the  coupling  of  a  clear  requirement  and  enforcement  of  that  requirement,  assess¬ 
ment  will  always  be  perceived  as  an  additional  task,  as  opposed  to  an  integral  part  of  project 
implementation . 

The  level  of  burden  related  to  conducting  project  assessment  activities  will  also  depend  on 
the  way  in  which  data  are  captured  and  stored.  The  process  of  collecting  and  storing  project 
assessment  information  manually  or  on  local  computers  will  be  difficult  to  maintain  over  time, 
particularly  when  HA  projects  are  handed  off  from  one  personnel  team  to  the  next.  Central¬ 
ized  data  collection  for  project  assessment  using  the  OHASIS  (or  another  relevant)  database 
could  ensure  the  consistent  and  easy  capture  of  data  through  clearly  defined  data  fields  and 
automated  reminders.  This  approach  would  also  ensure  that  assessment  data  across  all  HA 
projects  is  accessible  to  appropriate  personnel  for  conducting  higher-level  program-  or  country- 
level  assessments.  To  reduce  the  burden  of  HA  project-level  assessment  and  to  allow  generated 
data  to  inform  higher-level  assessments,  the  collection  of  project  assessment  information  as 
outlined  in  the  prototype  handbook  should  be  incorporated  within  current  and  future 
efforts  to  upgrade  and  improve  the  OHASIS  database. 

Without  information  captured  in  OHASIS  or  another  similar  system,  project-level  assess¬ 
ment  will  be  of  limited  use — particularly  beyond  the  individual  project  level.  DoD  will  need 
to  decide  whether  it  will  begin  to  require  project-level  assessment  using  the  handbook  before 
this  retrofitting  of  OHASIS  is  complete  or  whether  it  would  prefer  to  wait  until  the  system 
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is  redesigned  to  support  project  assessment  activities.  While  starting  sooner  may  be  beneficial 
from  the  perspective  of  training  and  instilling  best  practices  in  project  management,  the  poten¬ 
tial  for  perceived  additional  burden  and  limited  usefulness  is  higher  without  the  systems  sup¬ 
port  that  would  be  provided  by  a  redesigned  OHASIS  database.  To  the  extent  that  funds  are 
available,  the  capture  of  HA  project  assessment  information  as  outlined  in  the  handbook 
should  be  incorporated  into  OHASIS  before  HA  project  assessment  is  required  and  the 
requirement  is  enforced.  The  revised  handbook  reflects  the  implementation  of  this  recom¬ 
mendation.  That  is,  it  assumes  that  the  OHASIS  system  will  have  been  redesigned  and  in  place 
by  the  time  the  handbook  is  being  used  in  the  field. 

Given  the  limited  pilot-testing,  we  received  little  feedback  on  the  best  ways  to  organize 
and  format  the  handbook  to  ensure  its  most  effective  use  in  the  field.  One  major  revision  that 
was  considered  was  to  change  the  order  of  the  handbook’s  major  sections  by  placing  the  user’s 
guide  before  the  primer.  This  was  considered  in  light  of  one  user’s  statement  that  it  would  be 
better  if  the  instructive  (as  opposed  to  the  informative)  part  of  the  handbook  came  sooner.  This 
change,  while  understood  by  additional  reviewers,  introduced  confusion,  so  the  original  order 
was  maintained.  Additional  pilot  tests  would  help  inform  the  best  order  of  presentation  for  the 
wider  population  of  intended  handbook  users. 

Another  aspect  of  presentation  that  we  were  unable  to  address  adequately  was  the  physical 
format  of  the  handbook.  We  sent  the  handbook  draft  to  reviewers  and  pilot-testers  electroni¬ 
cally,  and  they  printed  it  out.  Although  there  are  several  different  handbook  formats  that  could 
be  tested  in  an  additional  round  of  pilot  tests,  based  on  the  feedback  that  we  received,  we  rec¬ 
ommend  a  letter-sized,  spiral-bound  handbook  with  a  flexible  protective  cover  and  a  CD 
insert  that  includes  an  electronic  version  of  the  handbook  and  all  handbook  worksheets. 
The  letter-sized  format  would  ensure  that  all  worksheets  are  readable  and  large  enough  for 
recording  data.  The  inclusion  of  a  manipulable  electronic  version  of  each  worksheet  will  allow 
users  to  download  these  files  onto  their  computers  and  use  them  for  each  HA  project  that  they 
implement.  The  electronic  worksheets  will  be  important  tools  for  recording  project  assessment 
data  in  the  field,  with  or  without  the  accompanying  redesigned  OHASIS  database.  If  OHASIS 
is  not  redesigned,  the  electronic  worksheets  will  also  serve  as  a  means  for  tracking  and  storing 
project  assessment  indicators  over  time. 

Despite  limited  opportunities  to  pilot-test  this  handbook,  we  believe  that  it  represents  an 
important  introduction  to  M&E  concepts  and  a  useful  guide  to  project  assessment  that  can 
and  should  be  used  beyond  OHDACA-funded  HA  projects.  Feedback  from  reviewers  and 
multiple  requests  to  use  the  handbook  for  training  seem  to  corroborate  this  view.  It  is  hoped 
that  a  final  handbook,  designed  with  a  specific  focus  in  mind,  will  be  used  more  broadly  by 
DoD  and  potentially  beyond.  To  support  this  broader  use  within  DoD,  in  particular,  the 
handbook  should  be  incorporated  into  the  formal  DoD  handbook  series.  This  would 
require  its  periodic  review  and  update  and  would  likely  encourage  its  use  for  training  and 
project-level  assessment,  both  across  DoD  and  in  other  interested  U.S.  government  agencies. 
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Introduction 


Background  and  Context 

The  U.S.  Department  of  Defense  (DoD)  defines  humanitarian  assistance  (HA)  as  “programs 
conducted  to  relieve  or  reduce  the  results  of  natural  or  manmade  disasters  or  other  endemic 
conditions  such  as  human  pain,  disease,  hunger,  or  privation  that  might  present  a  serious  threat 
to  life  or  that  can  result  in  great  damage  to  or  loss  of  property.”1  HA  has  long  been  a  function 
of  DoD,  and  HA  plays  an  important  role  in  current  military  operations.  HA  operations  are 
typically  carried  out  to  support  broader  strategic  goals  of  U.S  policy,  such  as  national  security 
related  to  reconstruction  and  stabilization,  and  DoD,  such  as  achieving  improved  access  and 
influence.  DoD  notes  that  HA  “provided  by  U.S.  forces  is  limited  in  scope  and  duration.  The 
assistance  provided  is  designed  to  supplement  or  complement  the  efforts  of  the  host  nation 
civil  authorities  or  other  U.S.  government  agencies  that  may  have  the  primary  responsibility 
for  providing  humanitarian  assistance.”2  DoD  HA  includes  activities  that  the  civilian  com¬ 
munity  would  consider  development.  Thus,  while  typically  short-term  in  nature,  they  should 
be  designed  with  a  longer-term  vision  so  they  are  consistent  (or  at  least  do  not  conflict)  with 
longer-term  development  objectives. 

DoD  wants  to  ensure  that  the  goals  and  objectives  of  its  HA  missions  are  met.  The  pur¬ 
pose  of  this  handbook  is  to  provide  a  practical,  easy-to-use,  step-by-step  guide  to  planning 
and  executing  DoD  HA  project  assessments.  It  was  designed  with  a  particular  focus  on  HA 
projects  funded  by  Overseas  Humanitarian,  Disaster,  and  Civic  Aid  (OHDACA)  appropria¬ 
tions,  but  it  should  be  applicable  to  the  assessment  of  other  assistance  projects  both  within  and 
outside  DoD. 

The  development  of  this  handbook  is  part  of  an  increased  emphasis  on  assessment  across 
DoD.  While  the  focus  of  this  handbook  is  assessment  at  the  HA  project  level,  it  was 
developed  so  that  the  information  collected  can  be  used  for  higher-level  program-  and 
strategic- level  assessments.  For  example,  assessments  related  to  individual  OHDACA  proj¬ 
ects  can  be  rolled  up  to  inform  the  achievement  of  overall  OHDACA  program  objectives  (ana¬ 
lyzed  via  a  more  comprehensive  program  assessment);  similarly,  OHDACA  program  assess¬ 
ments,  along  with  other  program  assessments,  can  be  rolled  up  to  inform  the  achievement  of 
national  strategic  objectives. 


1  U.S.  Joint  Chiefs  of  Staff,  Department  of  Defense  Dictionary  of  Military  and  Associated  Terms ,  Washington,  D.C., 
April  12,  2001,  as  amended  through  March  17,  2009. 

2  U.S.  Department  of  Defense,  Policy  Guidance  for  FY08  Overseas  Humanitarian  Assistance ,  September  19,  2007. 
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Reasons  for  Assessing  HA  Projects:  Why  Do  It? 

HA  project  assessment  will  usually  involve  the  use  of  scarce  staff  time  and  resources.  It  is  there¬ 
fore  important  to  understand  its  full  utility  before  engaging  in  it.  If  done  correctly,  project 
assessment  will  help  project  managers  to  ensure  that  their  projects  are  carried  out  as  intended 
and  produce  the  desired  results. 

Project  assessment  can  serve  many  functions  and  can  be  used  to  assess  project  outcomes 
in  both  the  short  and  long  terms.  In  the  short  term,  monitoring  the  performance  of  typically 
short  HA  projects  allows  project  managers  to  review  activities  and  make  midcourse  corrections 
as  well  as  ensure  that  subsequent  projects  take  advantage  of  any  lessons  learned.  For  longer, 
multiyear  projects,  performance  monitoring  during  project  execution  allows  project  managers 
to  identify  necessary  midcourse  corrections  that  will  allow  the  project  to  finish  on  time  and 
within  budget  and  produce  the  desired  results.  In  the  longer  term,  project  assessment  can  be 
used  to  document  the  outcomes  of  a  project  after  completion  and  the  sustainability  of  out¬ 
comes  over  time,  especially  the  capacity  of  the  host  nation  to  carry  on  activities  made  possible 
by  an  HA  project. 

In  general,  project  assessment  information  that  is  collected  will  be  contained  in  a  report 
(such  as  an  after-action  report)  associated  with  the  project.  Specific  measures,  or  indicators, 
are  gathered  over  time  and,  along  with  qualitative  interpretations,  become  the  basis  for  com¬ 
municating  up  the  chain  of  command  whether  and  how  the  project  met  its  specific  objectives. 

Future  resource  allocations  may  depend  on  past  project  achievements,  and  data  from 
project  assessments  can  guide  those  choices.  HA  project  planners  should  take  higher-order 
strategic  guidance  into  account  when  planning,  implementing,  and  assessing  projects,  since 
project-level  achievements  can  feed  into  the  achievement  of  higher-order  objectives  (e.g.,  at  pro¬ 
gram,  country,  theater,  and  combatant  command  [COCOM]  levels).  Thus,  HA  project  assess¬ 
ment  is  important  to  project  planners  and  managers  as  well  as  higher-level  decisionmakers. 


Purpose  and  Approach 

DoD’s  goal  in  using  monitoring  and  evaluation  (M&E)  or  assessment  is  to  ensure  that  its  stra¬ 
tegic  objectives  are  achieved,  from  the  tactical  up  to  the  strategic  level.  This  handbook  aims  to 
assist  in  that  process,  focusing  on  tactical-level  project  assessment  (i.e.,  project  monitoring)  and 
suggesting  that  the  collection  of  this  project-specific  assessment  information  can  contribute  to 
higher-level  program  and  strategy  assessments  (i.e.,  higher-order  evaluations). 

M&E  and  assessment  requirements  and  guidance  are  relatively  recent — but  have  been 
increasingly  emphasized — in  DoD.  This  means  that  some  people  responsible  for  project  over¬ 
sight  and  assessment  may  not  be  familiar  with  M&E  principles  and  the  practice  of  project 
assessment.  This  handbook  is  designed  to  provide  a  basic  understanding  of  M&E  concepts  and 
terms  and  to  facilitate  project  assessment.  It  was  designed  to  be  user-friendly,  focused  on  DoD 
priorities,  compliant  with  DoD  guidance,  and  compatible  with  guidance  from  and  current 
practice  in  relevant  agencies  beyond  DoD. 

Where  possible,  the  handbook  draws  on  widely  used  frameworks  from  relevant  civil¬ 
ian  programs.  For  example,  DoD  HA  work  often  complements  efforts  of  other  U.S.  govern¬ 
ment  (USG)  agencies,  most  notably  the  U.S.  Agency  for  International  Development  (USAID) 
and  the  U.S.  Department  of  State.  This  handbook  incorporates  relevant  M&E  terminology 
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and  constructs  used  by  USAID,  the  Department  of  State,  other  USG  agencies,  and  a  small 
number  of  nongovernmental  organizations  (NGOs).  Thus,  it  presents  a  real  opportunity  to 
both  standardize  HA  project  assessment  across  DoD  and  facilitate  interagency  commu¬ 
nication  when  DoD  collaborates  with  or  relies  on  other  agencies  to  measure  the  effectiveness 
of  HA  projects. 


Organization  and  Use  of  the  Handbook 

This  handbook  is  divided  into  two  parts:  Part  I,  “M&E  Primer,”  and  Part  II,  “Step-by-Step 
User’s  Guide  for  Project  Assessment.”  The  M&E  Primer  consists  of  three  chapters.  Chapter 
One  provides  a  general  overview  of  M&E  and  describes  basic  concepts  and  vocabulary  related 
to  the  project  life  cycle.  Chapter  Two  describes  planning  for  HA  project  assessment,  includ¬ 
ing  the  articulation  of  tactical-level  project  objectives  and  higher-order  program  and  strategic 
objectives  and  the  identification  of  appropriate  measures  of  performance  (MOPs)  and  mea¬ 
sures  of  effectiveness  (MOEs)  based  on  those  objectives.  Chapter  Three  provides  a  core  set  of 
required  indicators  to  be  used  across  all  OHDACA  projects,  as  well  as  required  indicators  for 
specific  project  types.  It  also  provides  guidance  on  selecting  additional  indicators  to  fit  specific 
project  needs  and  objectives.  Part  I  will  be  of  interest  mostly  to  users  who  are  new  to  the  con¬ 
cepts  of  monitoring  and  evaluation,  but  it  can  also  serve  as  a  refresher  for  more  seasoned  users 
or  as  a  reference  guide  for  any  user  assessing  a  project  using  the  user’s  guide  in  Part  II.  Part  II 
describes  the  execution  of  HA  project  assessments,  step  by  step,  from  start  to  finish.  For  users 
who  are  familiar  with  M&E  principles  and  project  assessment  practice,  Part  II  is  likely  all  that 
will  be  required  to  conduct  project  assessment  activities.  The  handbook  also  includes  detailed 
information  about  two  methodological  issues  that  are  relevant  to  project  monitoring:  commu¬ 
nity  consultation  (Appendix  A)  and  avoiding  bias  (Appendix  B).  It  concludes  with  a  glossary 
of  core  terms  that  are  relevant  to  M&E  (Appendix  C)  and  a  list  of  references. 

This  handbook  is  accompanied  by  a  CD  that  contains  both  an  electronic  version  of  the 
handbook  text  and  the  handbook  worksheets  in  an  editable  format.  The  worksheets  are  pre¬ 
sented  in  the  Step-by-Step  User’s  Guide,  along  with  instructions  for  completing  them.  Further 
details,  including  how  the  indicators  were  developed,  are  provided  in  Chapter  Three  of  the 
primer  (Part  I).  The  worksheets  should  be  used  during  project  planning  (i.e.,  the  “objective 
tree”  tool)  and  for  data  collection  throughout  the  course  of  HA  project  implementation  (to 
collect  baseline,  immediate,  and  follow-up  MOP  and  MOE  indicators).  Information  can  be 
entered  by  hand  onto  hard  copy  printouts  of  worksheets,  which  may  be  particularly  useful 
when  collecting  information  in  the  held,  and  then  keyed  in  electronically  later  to  local  proj¬ 
ect  team  computers  or  the  Overseas  Humanitarian  Assistance  Shared  Information  System 
(OHASIS). 
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M&E  Basics 


The  purpose  of  this  chapter  is  to  provide  a  basic  introduction  to  important  concepts  and  vocab¬ 
ulary  for  monitoring  and  evaluation  (M&E).  Throughout  this  handbook,  we  refer  mostly  to 
“assessment” — particularly  project  assessment.  Assessment  is  a  general  term  that  covers  the  con¬ 
cepts  of  both  “monitoring”  and  “evaluation,”  each  of  which  has  a  distinct  meaning  in  the  civil¬ 
ian  community.  Monitoring  typically  refers  to  individual  project  management  and  tracking  of 
inputs,  processes,  outputs,  and  outcomes,  whereas  evaluation  typically  refers  to  more  in-depth 
analysis  to  assess  higher-order  impacts  (e.g.,  achievement  of  higher-order  program  or  strategic 
objectives),  the  reasons  for  these  impacts,  or  other  analytic  questions.  Evaluation  is  typically 
conducted  across  a  variety  of  individual  projects,  at  the  program  or  even  a  more  strategic 
level,  and  is  often  done  by  a  third-party  evaluator.  These  distinctions  are  not  important  for 
U.S.  Department  of  Defense  (DoD)  humanitarian  assistance  (HA)  project-level  assessment, 
so  assessment  is  used  throughout  this  handbook  when  referring  to  HA  project-level  assessment. 
However,  many  of  the  concepts  and  terms  presented  in  this  chapter  are  those  used  in  the  civil¬ 
ian  world  to  refer  to  M&E. 

This  chapter  includes  a  diagram  that  illustrates  the  steps  in  the  HA  project  cycle  and  the 
steps  in  a  project  assessment  that  can  be  used  to  measure  performance  and  effectiveness.  This 
chapter  reviews  the  project  and  assessment  cycles  and  introduces  basic  concepts  and  terms 
related  to  each,  using  examples  from  relevant  DoD  HA  projects  around  the  world  to  illustrate 
the  main  points.  This  chapter  also  links  the  key  DoD  terms  measure  of  performance  (MOP)  and 
measure  of  effectiveness  (MOE)  with  analogous  terms  used  in  the  civilian  world.  These  linkages 
are  important  to  facilitate  communication  and  collaboration  between  in-country  HA  project 
managers  and  managers  of  similar  projects  in  other  U.S.  government  (USG)  agencies  or  non¬ 
governmental  organizations  (NGOs). 


The  Humanitarian  Assistance  Project  Cycle 

The  main  components  of  the  project  cycle  are  shown  in  Figure  1.1.  During  the  planning 
phase  of  a  project,  a  planner  considers  local  needs  and  integrates  the  goals  of  key  stakehold¬ 
ers  (e.g.,  local  partners  and  members  of  the  country  team — the  USG  team  in  a  country — 
including  DoD  representatives)  with  higher-level  strategic  objectives  (e.g.,  from  theater  cam¬ 
paign  plans)  to  determine  the  best  project  to  undertake  and  identify  the  reasons  for  under¬ 
taking  it.  The  project  proposal  is  developed  in  detail,  including  project  objectives,  expected 
outcomes  (i.e.,  results  from  or  effects  of  the  project  and  its  outputs),  planned  outputs 
(i.e.,  products  that  result  from  the  project),  project  activities,  and  required  inputs  (e.g.,  stake- 
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Figure  1.1 
Project  Cycle 
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holder  buy-in,  funding,  materials,  land).  These  project  elements  are  used  to  develop  a  project 
timeline,  budget,  and  assessment  plan  as  part  of  the  project  proposal  (nomination). 

Assuming  that  the  project  is  approved  and  resources  are  provided,  the  project  is  executed. 
The  start-up  phase  entails  all  the  preparation  required  to  execute  the  project,  including  assess¬ 
ing  local  conditions  on  the  ground  and  consideration  of  the  need  for  contractors.  The  execu¬ 
tion  phase  of  the  project  is  when  the  actual  work  is  done:  when  the  school  is  built,  the  well 
dug,  the  clinic  refurbished,  and  so  on.  Figure  1.1  breaks  down  the  execution  phase  into  three 
subcomponents: 

•  Inputs  refer  to  the  broad  set  of  resources  required  to  get  the  project  done  (e.g.,  fund¬ 
ing,  material,  human  resources)  and  pre-planning  steps  that  should  be  in  place  prior  to 
undertaking  project  activities  (e.g.,  project-related  plans).  For  a  well  project,  inputs  might 
include  the  land  on  which  the  well  is  to  be  dug,  the  machinery  necessary  to  do  the  drill¬ 
ing,  the  materials  necessary  to  build  an  appropriate  cover  for  the  well,  and  the  project 
assessment  plan  for  testing  project  performance  and  effectiveness. 

•  Processes  are  the  activities  that  must  be  undertaken  to  achieve  the  project  objectives.  These 
might  include  working  with  the  community  to  determine  the  best  site  for  the  well,  drill¬ 
ing  the  well,  and  building  appropriate  covers  or  shelters  for  the  well. 

•  Outputs  are  the  direct,  often  physical,  results  of  the  processes.  Examples  of  outputs  include 
a  well  that  meets  some  set  of  specifications  or  the  number  of  people  trained  to  maintain 
the  well  or  monitor  water  quality.  Outputs  are  different  from  outcomes,  as  explained  next. 

In  the  project  cycle,  project  inputs  feed  into  processes  that  lead  to  project  outputs.  Those 
outputs  (e.g.,  a  well  is  built,  a  school  is  refurbished,  people  are  trained)  are  expected  to  have 
short-  and  long-term  outcomes  in  the  community.  Short-term  (immediate)  outcomes  are  those 
that  are  expected  immediately  after  project  completion  or  shortly  thereafter;  long-term  (sus¬ 
tained)  outcomes  are  maintained  or  manifest  one  or  more  years  after  project  completion  and 
are  usually  observed  during  the  project  follow-up  stage.  In  the  ideal  project,  outputs  lead  to 
both  immediate  and  sustained  outcomes,  such  as  improved  drinking  water  quality  and  other 
improvements  for  the  community,  capacity-building  for  local  partners,  and  increased  military 
access  to  and  influence  in  the  community. 
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Humanitarian  Assistance  Project  Assessment 

Measuring  performance  and  effectiveness  through  project  assessment  is  integral  to  the  success 
of  any  HA  project.  DoD  defines  assessment  as  “a  continuous  process  that  measures  the  overall 
effectiveness  of  employing  joint  force  capabilities  during  military  operations”  and  “determina¬ 
tion  of  the  progress  toward  accomplishing  a  task,  creating  an  effect,  or  achieving  an  objective.”1 

The  overall  goals  of  project  assessment  are  to  ensure  that  the  project  is  executed  according 
to  plan,  enable  midcourse  corrections  for  projects  long  enough  or  large  enough  to  warrant  mea¬ 
surements  during  project  execution,  provide  information  about  the  immediate  and  sustained 
outcomes  from  the  project,  provide  information  about  project  contributions  to  higher-order 
strategic  objectives,  and  provide  feedback  that  will  be  useful  in  the  design  and  implementation 
of  subsequent  projects. 

Planning  for  project  assessment  is  an  integral  part  of  project  planning  and  management 
and  should  be  undertaken  as  part  of  the  project  design  phase.  While  a  project  is  being  devel¬ 
oped,  project  planners  should  think  about  the  results  they  wish  to  achieve,  the  processes  that 
can  be  used  to  help  ensure  the  desired  results,  and  the  indicators  that  can  be  used  to  measure 
results.  Project  assessment  must  incorporate  every  aspect  of  the  project  cycle,  including  inputs, 
processes  (activities),  outputs,  and  outcomes,  and  should  be  based  on  both  project-specific 
objectives  and  the  project’s  contribution  to  higher-order  strategic  objectives.  Figure  1.2  shows 
the  components  of  project  assessment  as  they  relate  to  the  project  cycle. 

Figure  1.2 

Measuring  Performance  and  Effectiveness:  Links  to  the  Project  Cycle 
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1  U.S.  Joint  Chiefs  of  Staff,  Joint  Operations ,  Joint  Publication  3-0,  September  17,  2006,  incorporating  change  1, 
February  13,  2008. 
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The  steps  for  project  assessment  planning  are  shown  on  the  left  side  of  the  two  panels  in 
Figure  1.2.  The  most  critical  part  of  the  assessment  planning  phase  is  to  identify  the  relevant 
MOEs  (that  is,  measures  for  outcomes  that  will  indicate  that  the  desired  results  have  been 
achieved)  and  MOPs  (that  is,  measures  of  inputs,  processes,  and  outputs  indicating  whether 
the  project’s  performance  is  on  track).  In  addition,  it  is  important  to  develop  a  plan  for  data 
collection  throughout  the  project  cycle  (including  a  timeline,  methods,  and  the  resources 
required)  and  a  plan  for  data  use  (how  data  will  be  reported,  to  whom,  and  how  such  data 
might  be  used). 

An  important  component  of  the  data  collection  plan  is  a  plan  to  collect  information  on 
desired  outcomes  both  before  and  after  project  execution  so  that  changes  in  outcomes  of  inter¬ 
est  can  be  documented.  For  example,  the  desired  outcome  for  refurbishing  an  existing  school 
or  for  building  a  new  school  may  be  enhanced  access  to  school  among  children  within  a  par¬ 
ticular  geographic  area  or  increased  numbers  of  children  enrolled  in  school.  In  this  example, 
to  confirm  that  the  desired  outcome  is  achieved  once  project  execution  is  complete,  it  is  essen¬ 
tial  to  measure  the  distance  or  time  that  children  in  the  population  of  interest  must  travel  to 
attend  school  or  the  number  of  children  enrolled  in  school  before  a  school  is  refurbished  or  a 
new  school  is  built.  This  is  called  a  baseline  measure  of  the  outcome  indicator.  The  collection 
of  baseline  indicators  should  be  carried  out  at  project  start-up,  just  prior  to  the  beginning  of 
project  execution,  so  that  appropriate  comparisons  can  be  made  upon  project  completion. 

The  two  primary  DoD  terms  related  to  M&E  are  measure  of  performance  (MOP)  and 
measure  of  effectiveness  (MOE).  As  described  later  in  this  chapter,  these  terms  have  formal 
definitions  in  the  military  (see  Tables  1.1  through  1.4).  MOPs  and  MOEs  represent  information 
collected  to  document  and  communicate  whether  the  project  manager  is  on  track  with  project 
execution  and  has  achieved  the  desired  results.  Another  word  for  such  measures  is  indicator. 
The  joint  Department  of  State— U.S.  Agency  for  International  Development  (USAID)  M&E 
dictionary  defines  an  indicator  as  a  “quantitative  or  qualitative  variable  that  provides  reliable 
means  to  measure  a  particular  phenomenon  or  attribute.”2  In  the  context  of  this  handbook,  an 
indicator  specifies — in  words  or  numbers — a  level  of  objective  achievement,  measured  in  terms 
of  outcomes  (MOEs)  or  tangible  elements  associated  with  project  execution  (MOPs). 

Measures  of  Performance 

A  MOP  is  “a  criterion  used  to  assess  friendly  actions  tied  to  measuring  task  accomplishment.”3 
That  is,  an  MOP  is  an  indicator  of  progress  during  the  implementation  of  a  project.  MOPs  are 
collected  during  the  execution  phase  of  the  project  cycle  (see  Figure  1.2). 

Often,  after  the  relevant  resources  for  a  project  have  been  acquired,  the  military  turns  to 
local  contractors  or  other  local  partners  to  execute  the  proposed  HA  project.  In  these  cases,  it 
might  be  particularly  important  to  monitor  whether  the  project  is  going  to  be  finished  on  time, 
within  budget,  and  so  on.  These  are  examples  of  performance  indicators  or  MOPs.  Today, 
many  in-country  HA  project  managers  are  already  measuring  the  performance  of  their  proj¬ 
ects  using  the  “on-time-and-within-budget”  criteria.  For  example,  midcourse  corrections  are 
often  made  based  on  information  about  progress  relative  to  proposed  timelines  and  budgets. 


2  U.S.  Agency  for  International  Development,  Planning  and  Performance  Management  Unit,  Office  of  the  Director  of 
Foreign  Assistance,  Glossary  of  Evaluation  Terms,  March  25,  2009. 

3  U.S.  Joint  Chiefs  of  Staff,  2008. 
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When  the  project  is  completed,  other  (output)  MOPs  might  focus  on  whether  the  item  built, 
produced,  or  delivered  meets  a  particular  set  of  standards. 

As  shown  in  Figure  1.3,  three  types  of  indicators  are  associated  with  MOPs.  The  figure 
shows  input,  process,  and  output  indicators  subsumed  under  the  umbrella  term  measures  of 
performance  to  highlight  the  connection  between  the  concepts  presented  in  the  project  cycle 
diagram  (Figure  1.1)  and  the  concepts  presented  in  the  HA  guidance. 

Input  indicators  are  criteria  used  to  assess  access  to  all  necessary  resources  for  project 
completion  and  pre-planning  steps  that  should  be  in  place  before  project  activities  are  under¬ 
taken.  For  example,  “Land  was  provided  on  time  for  well  construction,”  or  “Land  was  pur¬ 
chased  with  funds  provided  by  local  NGOs.”  Process  indicators  are  the  criteria  used  to  assess 
progress  associated  with  project  activities.  Examples  include  whether  a  project  was  completed 
within  its  established  deadline  and  whether  the  project  was  implemented  in  conjunction  with 
host-nation  counterparts.  Process  indicators  can  also  be  used  to  communicate  qualitative  data 
about  the  project.  For  example,  in  projects  that  are  long  enough  (e.g.,  multiyear)  or  large 
enough  to  warrant  measurement  of  process  indicators  during  project  execution,  solutions  to 
problems  may  be  discovered  along  the  way.  Conversely,  problems  may  be  uncovered  after  a 
(typically  shorter)  project  is  completed.  Either  way,  detailed  information  can  provide  lessons 
learned  for  the  project  planner  and  for  others  who  undertake  similar  projects  in  the  future. 

This  information  might  be  communicated  in  a  narrative  way,  using  short  paragraphs  to 
explain  the  problem,  the  solution,  and  the  resolution.  Output  indicators  are  criteria  used 
to  assess  the  direct  products  of  the  project  activities.  Output  indicators  can  be  used  to  com¬ 
municate  that  the  work  was  completed  (e.g.,  well  built  by  x  date)  and  that  specifications  were 
met  (e.g.,  well  meets  specified  local,  national,  or  international  water  and  sanitation  standards). 

Table  1.1  summarizes  the  definitions  of  MOPs,  presents  different  types  of  MOP  indica¬ 
tors,  and  provides  a  simple  example  of  each. 

Other  terms  have  been  used  to  describe  the  same  general  concepts  captured  by  the  term 
measure  of  performance.  Key  M&E  terms  related  to  MOPs  are  shown  in  Table  1.2.  For  example, 
an  MOP  is  sometimes  referred  to  as  a  “measure  of  process”  or  “measure  of  progress”  because 
it  is  an  indicator  associated  with  the  execution  of  humanitarian  assistance  projects.  The  terms 
performance  indicator  and  process  indicator ,  used  by  the  United  Nations,  the  Organisation  for 
Economic  Co-Operation  and  Development,  USAID,  the  Department  of  State,  and  others,  are 
related  terms,  as  noted  in  Table  1.2.  What  all  these  definitions  highlight  is  the  importance  of 
ensuring  that  the  relevant  data  are  collected  and  the  project  is  executed  on  time,  within  budget, 
and  without  problems,  or  that  problems  encountered  are  resolved. 

Figure  1.3 

Terms  Associated  with  Measures  of  Performance 
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Table  1.1 

Measures  of  Performance:  Types,  Definitions,  and  Examples 


Measures  of  Performance 

Criteria  used  to  assess  friendly  actions  tied  to  measuring  task  accomplishment.  Also  called  MOPs.  MOPs  are 
generally  quantitative,  but  they  can  also  apply  qualitative  attributes  to  task  accomplishment.3 

Type 

Input  indicator 

Process  indicator 

Output  indicator 

Definition 

Criterion  used  to  assess  pre¬ 
project  planning  steps  that 
should  be  completed  prior  to 
project  start  and  access  to  all 
necessary  resources  for  project 
completion 

Criterion  used  to  assess 
progress  of  project  activities 

Criterion  used  to  assess  direct 
products  of  project  activities 

Example 

Adequate  staff  to  complete 
the  project 

Activities  completed  within  a 
certain  time  frame 

Well  built  to  specified  local, 
national,  or  international 
standards 

a  U.S.  Joint  Chiefs  of  Staff,  2008. 


Measures  of  Effectiveness 

An  MOE  is  defined  by  DoD  as  “a  criterion  used  to  assess  changes  in  system  behavior,  capa¬ 
bility,  or  operational  environment  that  is  tied  to  measuring  the  attainment  of  an  end  state, 
achievement  of  an  objective,  or  creation  of  an  effect.”4  MOEs  help  project  managers  commu¬ 
nicate  how  well  a  project  objective  was  achieved,  the  impact  of  the  project  activity  on  the  com¬ 
munity,  the  extent  to  which  the  project  improved  local  capacity,  and  so  on.  MOEs  measure  the 
outcomes  of  the  project.  Thus,  as  shown  earlier  in  Figure  1.2,  outcome  indicators  representing 
MOEs  must  be  collected  at  various  points  in  time  to  assess  changes  in  desired  outcomes  and 
the  sustainability  of  those  outcomes.  Figure  1.4  shows  the  sequencing  of  data  collection  that  is 
generally  required  to  obtain  MOEs. 

Once  a  project  has  been  approved,  but  before  it  is  executed,  it  is  important  to  obtain — 
from  existing  sources,  such  as  USAID  or  local  NGOs — or  to  collect  information  to  determine 
the  baseline  against  which  short-  and  long-term  outcomes  of  the  project  can  be  compared. 
This  stage  is  represented  by  the  first  box  in  Figure  1.4.  Outcome  indicators  (or  MOEs)  collected 
at  baseline  (project  start-up)  provide  information  on  the  starting  point  of  any  project:  Under 
what  circumstances  in  the  local  area  did  the  project  begin?  The  same  outcome  indicators  collected 
at  baseline  are  collected  again  after  the  project  is  completed,  so  that  any  changes  caused  by  the  project 
can  be  assessed.  Outcome  indicators  are  often  collected  again  one  or  more  times  in  the  future  to  assess 
the  sustainability  of  the  project  results  over  time. 

At  project  completion,  an  immediate  assessment  of  outcomes  is  important.  This  assess¬ 
ment  should  determine  whether  the  project  achieved  its  specific  objectives  and  contributed 
to  higher-level  goals.  It  is  important  to  note  that  the  methods  proposed  here  may  not  allow 
rigorous  attribution  of  outcomes  to  project  activities,  but  they  nonetheless  fulfill  the  needs  of 
in-country  FI  A  project  managers.  Comparing  outcome  indicators  immediately  post-project 
to  the  baseline  data  collected  before  the  project  started  will  help  in  communicating  the  kind 
of  changes  the  project  has  effected  in  the  community.  Here,  the  point  is  to  assess  the  project 
soon  after  its  completion,  using  measurable  indicators  to  determine,  for  example,  the  extent  to 


4  U.S.  Joint  Chiefs  of  Staff,  Department  of  Defense  Dictionary  of  Military  and  Associated  Terms,  Washington,  D.C., 
April  12,  2001,  as  amended  through  March  17,  2009. 
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Table  1.2 

Key  M&E  Terms  Related  to  Measures  of  Performance 


Term 


Definition 


Indicator 


Inputs 


Outputs 


Performance 


1.  A  quantitative  or  qualitative  variable  that  provides  a  reliable  means  to  measure  a 
particular  phenomenon  or  attribute.3 

2.  Quantitative  or  qualitative  factor  or  variable  that  provides  a  simple,  reliable  basis  for 
assessing  achievement,  change,  or  performance.  A  unit  of  information  measured  over 
time  that  can  help  show  changes  in  a  specific  condition.  A  given  goal  or  objective  can  have 
multiple  indicators.13 

1.  Resources  provided  for  program  implementation.  Examples  include  money,  staff,  time, 
facilities,  equipment,  and  plans.3 

2.  The  financial,  human,  material,  technological,  and  information  resources  provided  by 
stakeholders  (i.e.,  donors,  program  implementers,  and  beneficiaries)  that  are  used  to 
implement  a  development  intervention. c 

This  handbook  also  adds  pre-planning  steps  that  should  be  in  place  as  inputs  before  a 
project  commences. 

1.  The  products,  goods,  and  services  that  result  from  an  intervention.3 

2.  A  final  product  or  service  delivered  by  a  program  or  project  to  beneficiaries,  such  as 
goods,  services,  training,  or  facilities  that  a  program  is  expected  to  produce  to  achieve  its 
expected  objectives. d 

Degree  to  which  a  development  intervention  or  development  partner  operates  according  to 
specific  criteria,  standards,  or  guidelines  or  achieves  results  in  accordance  with  stated  goals 
or  plans. e 


Performance  or  1 .  A  particular  characteristic  or  dimension  used  to  measure  intended  changes.  Performance 
process  indicator  indicators  are  used  to  observe  progress  and  measure  actual  results  against  expected 
results.3 

This  definition  differs  slightly  from  the  DoD  definition  in  that  the  USAID/Department  of 
State  definition  includes  results.  It  spans  the  military's  definitions  of  both  MOP  and  MOE. 

2.  Specific  statistics  chosen  because  they  provide  valid,  practical,  and  comparable  measures 
of  progress  or  level  of  change  toward  achieving  expected  results  in  a  given  period. 

Used  to  measure  the  extent  to  which  goals  have  been  achieved.  Indicators  correspond 
to  the  expected  accomplishment  and  are  used  to  measure  performance.  One  expected 
accomplishment  can  have  multiple  indicators.01 

Related  terms:  achievement  indicator,  expected  accomplishment,  performance  measure 


NOTE:  Definitions  are  paraphrased. 

3  U.S.  Agency  for  International  Development,  Planning  and  Performance  Management  Unit,  Office  of  the 
Director  of  Foreign  Assistance,  Glossary  of  Evaluation  Terms,  March  25,  2009. 

b  International  Fund  for  Agricultural  Development,  "Annex  A:  Glossary  of  M&E  Concepts  and  Terms,"  in 
Managing  for  Impact  in  Rural  Development:  A  Guide  for  Project  M&E,  2002. 

c  United  Nations  Population  Fund,  Division  for  Oversight  Services,  "Tool  Number  1:  Glossary  of  Planning, 
Monitoring,  and  Evaluation  Terms,"  in  Programme  Manager's  Planning,  Monitoring  and  Evaluation  Toolkit, 
March  2004. 

d  United  Nations  Monitoring,  Evaluation  and  Consulting  Division,  Glossary  of  Monitoring  and  Evaluation  Terms, 
August  2006. 

e  Organisation  for  Economic  Co-Operation  and  Development,  Development  Assistance  Committee,  Glossary  of 
Key  Terms  in  Evaluation  and  Results  Based  Management,  2002. 


which  local  access  to  sources  of  drinking  water  and  water  quality  have  improved  and  whether 
trained  members  of  the  community  maintain  the  well  or  facility  that  was  built  or  renovated  as 
part  of  the  project. 

Outcome  indicators  at  follow-up  are  measured  to  assess  whether  project  outcomes  have 
been  sustained  over  a  longer  period.  To  what  extent  do  community  members  still  have  access 
to  clean  drinking  water  from  the  well?  How  many  are  attending  the  school  or  seeking  health 
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Figure  1.4 

Sequencing  of  Data  Collection  for  Measures  of 
Effectiveness 
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services  at  the  clinic?  Are  capacities  that  were  built  during  project  execution  still  evident? 
Comparing  outcome  indicators  at  follow-up  to  those  collected  at  baseline  and  immediately 
after  project  completion  will  help  communicate  any  lasting  changes  achieved  (i.e.,  MOEs)  in 
a  community. 

To  summarize  the  preceding  discussion,  Table  1.3  provides  a  definition  of  MOEs,  the 
general  sequencing  for  data  collection  of  MOE  outcome  indicators,  a  description  of  how  they 
are  used  at  different  points  in  time,  and  an  example  of  each. 

The  DoD’s  use  of  the  term  effectiveness  is  related  to  the  M&E  terms  outcomes ,  effects, 
results,  and  impacts,  as  used  by  the  Department  of  State,  USAID,  the  United  Nations,  and 
other  organizations.  Formal  definitions  for  these  M&E  terms  are  presented  in  Table  1.4.  Mea¬ 
sures  of  effectiveness  (MOEs)  is  the  term  DoD  uses  to  communicate  project  success  in  reports  or 
other  communications  aimed  at  explaining  how  well  project  managers  are  doing. 

Table  1.4  does  not  include  all  the  important  terms  that  people  use  when  talking  about 
M&E.  These  terms  are,  however,  the  most  important  terms  used  by  civilian  agencies  and  orga¬ 
nizations  that  may  have  some  connection  to  DoD  HA  projects.  There  are  many  other  terms 
that  HA  planners  might  come  across  when  developing  a  project  assessment  plan  or  writing  up 
an  assessment  report  in  coordination  with  other  groups.  The  glossary  at  the  end  of  this  hand- 


Table  1.3 

Measures  of  Effectiveness:  Types,  Definitions,  and  Examples 


Measures  of  Effectiveness 

Criteria  used  to  assess  changes  in  system  behavior,  capability,  or  operational  environment  that  are  tied  to 
measuring  the  attainment  of  an  end  state,  achievement  of  an  objective,  or  creation  of  an  effect.  Also  called 
MOEs.  These  measures  are  typically  more  subjective  than  MOPs  and  can  be  crafted  as  either  qualitative  or 
quantitative.  Quantitative  measures  can  reflect  a  trend  or  show  progress  toward  a  measurable  threshold.3 


Data  collection  point 

Baseline 

Immediately  post-project 

Follow-up 

Description 

Criterion  to  determine 
the  original  state  of  the 
community 

Criterion  to  determine  the 
immediate  effects  of  a 
project  on  the  community 

Criterion  to  determine  the 
effects  of  a  project  on  the 
community  after  a  specified 
length  of  time 

Example 

Water  quality  before  the 
well  was  built 

Water  quality  after  the  well 
is  built  compared  to  before 
well  was  built 

Water  quality  after  one  year 
compared  to  just  after  the 
well  was  built  and  to  before 
the  well  was  built 

a  U.S.  Agency  for  International  Development,  2009. 
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Table  1.4 

Key  M&E  Terms  Related  to  Measures  of  Effectiveness 


Term 


Definition 


Baseline 


Effect 


Impact 


Indicator 


Outcome 


Result 


Information  collected  before  or  at  the  start  of  a  project  or  program  that  provides  a  basis  for 
planning  or  assessing  subsequent  progress  and  impact.3 

Intended  or  unintended  change  that  results  directly  or  indirectly  from  an  intervention.3 
Related  terms:  result,  outcome,  impact 

1 .  A  result  or  effect  that  is  caused  by  or  attributable  to  a  project  or  program.  Often  used  to 
refer  to  the  higher-level  effects  of  a  program  that  occur  in  the  medium  or  long  term  and 
can  be  intended  or  unintended,  positive  or  negative.3 

2.  The  overall  effect  of  achieving  specific  results.  In  some  situations,  it  includes  changes — 
planned  or  unplanned,  positive  or  negative,  direct  or  indirect,  primary  or  secondary — that 
a  program  or  project  helped  to  bring  about.  It  could  also  connote  the  maintenance  of  a 
current  condition,  assuming  that  the  condition  is  favorable.  Impact  is  the  longer-term  or 
ultimate  effect  attributable  to  a  program  or  project,  in  contrast  to  outputs  or  expected 
accomplishments,  which  are  shorter-term  effects.13 

Related  terms:  effect 

1.  A  quantitative  or  qualitative  variable  that  provides  a  reliable  means  to  measure  a 
particular  phenomenon  or  attribute.3 

2.  Quantitative  or  qualitative  factor  or  variable  that  provides  a  simple,  reliable  basis  for 
assessing  achievement,  change,  or  performance.  A  unit  of  information  measured  over 
time  that  can  help  show  changes  in  a  specific  condition.  A  given  goal  or  objective  can  have 
multiple  indicators. b 

1 .  A  result  or  effect  caused  by  or  attributable  to  a  project,  program,  or  policy.  Often  used  to 
refer  to  more  immediate  and  intended  effects.3 

Outcome  indicators  would  directly  measure  these  outcomes. 

2.  The  intended  or  achieved  short-  and  medium-term  effects  of  an  intervention's 
outputs,  usually  requiring  the  collective  effort  of  partners.  Outcomes  represent  changes 
in  development  conditions  that  occur  between  the  completion  of  outputs  and  the 
achievement  of  impact. c 

Related  terms:  effect,  impact,  output,  result 

1.  The  intended  (or  unintended)  output,  outcome,  or  impact.3 

USAID  and  the  Department  of  State  use  the  term  "performance  indicator"  to  identify 
the  direct  measures  of  a  result.  That  definition  is  broader  than  the  term  "measure  of 
performance"  and  includes  some  aspects  of  MOE. 

2.  The  measurable  output,  outcome,  or  impact  (intended  or  unintended,  positive  or 
negative)  of  a  development  intervention.13  (Emphasis  added.) 


NOTE:  Definitions  are  paraphrased. 

3  U.S.  Agency  for  International  Development,  2009. 
b  International  Fund  for  Agricultural  Development,  2002. 
c  United  Nations  Population  Fund,  2004. 


book  provides  a  more  complete  set  of  M&E-related  terminology  based  on  glossaries  or  diction¬ 
aries  developed  by  the  Department  of  State,  the  United  Nations,  USAID,  and  others. 


Pulling  It  All  Together 

Table  1.5  summarizes  the  information  presented  in  this  chapter  by  linking  elements  of  the 
project  cycle  to  corresponding  assessment  elements  (MOPs,  MOEs,  and  the  specific  types  of 
indicator  in  each)  and  the  questions  addressed  by  each  type  of  indicator. 
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Table  1.5 

Measuring  Performance  and  Effectiveness:  Summary 


Project  Phase 

Step  in 
Project  Cycle 

Step  in  Project 
Assessment  Cycle 

Indicators  Question  Answered 

Pre-Execution 

Start-up 

Baseline 

measurement 

Outcome  indicators:  What  are  the  conditions  before  the 
Baseline  project  begins? 

Execution 

Inputs 

Process 

Measuring 

performance 

Input  indicators  Were  required  resources  available? 

Process  indicators  Were  activities  completed  on  time 
and  within  budget? 

Outputs 

Output  indicators  What  direct,  tangible  products  or 
services  did  the  project  deliver? 

Completion 

Immediate 

outcomes 

Measuring 

effectiveness 

Outcome  indicators:  Did  the  project  achieve  its 
Immediately  objectives? 

post-project 

Follow-up 

Sustained 

outcomes 

Outcome  indicators:  Are  the  changes  produced  by  the 
Follow-up  project  still  evident? 

Now  that  the  basics  of  M&E  have  been  covered  in  terms  of  concepts  and  definitions, 
Chapter  Two  uses  a  practical  example  to  address  how  to  think  about  and  plan  for  project 
assessment. 


CHAPTER  TWO 


Planning  for  Project  Assessment 


Project  assessment  captures  key  aspects  of  performance  and  effectiveness  for  HA  projects  based 
on  the  objectives  that  the  project  seeks  to  achieve.  Once  a  planner  has  a  good  understanding 
of  the  needs  to  be  addressed  (e.g.,  unmet  community  needs),  a  project  can  be  designed.  Project 
design  includes  identifying  project  objectives,  the  related  desired  outcomes  and  outputs,  and 
required  activities  and  inputs.  Once  desired  outcomes  and  outputs  and  required  activities 
and  inputs  are  identified,  indicators  should  be  developed  to  measure  each  (i.e.,  MOPs  to  mea¬ 
sure  inputs,  activities,  and  outputs  and  MOEs  to  measure  outcomes)  so  that  progress  toward 
objective  achievement  can  be  tracked. 

As  will  be  discussed  in  Chapter  Three,  DoD  has  established  a  set  of  required  core  indica¬ 
tors  for  use  in  all  HA  projects,  as  well  as  indicators  related  to  each  of  the  major  types  of  HA 
projects.  These  required  indicators  cover  various  aspects  of  project  management  and  higher- 
order  objectives  as  stipulated  by  the  HA  policy  guidance;  they  also  address  general  objectives 
for  the  most  typical  HA  project  types.  They  will  allow  for  the  development  of  a  comprehen¬ 
sive,  comparable  data  set  across  all  HA  projects  and,  thus,  program-level  and  other  relevant 
assessments. 

Aside  from  these  standardized  indicators,  a  project  planner  may  need  to  develop  addi¬ 
tional  MOP  and  MOE  indicators — or  use  relevant  existing  indicators  from  USAID  or  other 
sources — to  measure  progress  toward  project-specific  objectives.  This  occurs  when  a  project  has 
unique  objectives  not  covered  by  the  required  indicators  or  when  it  does  not  fall  under  one  of 
the  five  project  types  for  which  indicators  have  been  developed. 

This  chapter  addresses  how  to  plan  for  project  assessment — that  is,  thinking  through 
project-specific  objectives  and  developing  appropriate  additional  MOPs  and  MOEs  (as  appli¬ 
cable).  A  realistic  scenario  of  an  HA  project  assessment  is  that  someone  arrives  at  post  and 
becomes  responsible  for  a  project  that  was  completed  recently  or  even  a  year  earlier  by  a  prede¬ 
cessor.  The  project  may  or  may  not  have  a  project  assessment  plan,  but  the  new  staff  member 
is  responsible  for  post-project  outcome  assessment.  Another  potential  scenario  is  that  the  new 
person  arrives  at  post  and  assumes  responsibility  for  a  project  that  was  nominated  by  the  pre¬ 
decessor.  The  project  has  been  approved,  and  the  new  person  is  asked  by  the  combatant  com¬ 
mand  (COCOM)  HA  manager  to  provide  a  project  assessment  plan  that  includes  specific 
MOEs.  Yet  another  scenario  is  that  the  person  is  beginning  to  think  about  a  project  nomina¬ 
tion  and  needs  to  develop  both  the  project  plan  and  the  associated  plan  for  project  assessment, 
which  means  determining  the  appropriate  MOEs  and  MOPs.  This  chapter  describes  the  bare 
minimum  that  must  be  done  to  determine  the  MOEs  and  MOPs.  It  uses  the  example  of  build¬ 
ing  a  village  well,  a  typical  HA  project,  for  illustration.  To  this  end,  the  chapter  introduces  a 
tool  that  can  be  used  to  guide  planners  and  project  managers  to  identify  project-specific  objec- 
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tives  and  select  related  MOPs  and  MOEs  by  examining  the  causes  and  effects  of  an  HA  project 
in  a  systematic  way. 


How  to  Determine  Measures  of  Effectiveness 

What  Objective  Is  the  Project  Intended  to  Achieve? 

When  feasible,  it  is  important  to  first  determine  the  project’s  objective,  or  desired  result,  before 
deciding  what  the  project  itself  will  be.  The  project  should  be  designed  or  determined  based 
on  the  objectives  to  be  achieved.  The  decision  to  build  a  well  is  important,  but  it  is  more  impor¬ 
tant  to  understand  the  reasons  for  building  the  well.  The  well  is  not  an  end  in  itself;  it  is  a  means 
to  achieve  a  desired  result  or  set  of  results.  MOEs  are  designed  to  measure  such  results.  The 
intended  objectives  must  be  clearly  stated  so  that  appropriate  MOEs  can  be  identified. 

Some  reasons  that  a  well  might  be  built  with  Overseas  Humanitarian,  Disaster,  and  Civic 
Aid  (OHDACA)  support: 

•  so  the  community  has  clean  water  and  is  healthier 

•  so  girls  can  spend  less  time  collecting  water  and  have  more  time  to  go  to  school 

•  so  women  can  spend  less  time  collecting  water  and  have  more  time  to  earn  income. 

These  are  examples  of  the  kinds  of  objectives  to  achieve  a  strategy  of  “sustained  positive 
impact  on  the  civilian  population.”  But  there  are  other  factors  that  must  be  addressed  for  a  well 
to  have  these  positive  results.  In  the  case  of  these  three  examples,  the  well  must  be  maintained 
for  good  water  quality,  there  must  be  schools  that  admit  girls,  and  there  must  be  productive 
ways  for  women  to  make  money. 

Beyond  project-level  objectives  of  sustained  positive  impact,  OHDACA  policy  guidance 
describes  some  higher-order  reasons  to  carry  out  an  HA  project,  or  providing  an  “indirect  ben¬ 
efit  to  USG  security  interests.”  Some  examples  of  such  higher-order  strategic  objectives  include 
the  following: 

•  Enhance  the  legitimacy  of  the  host  nation. 

•  Provide  evidence  of  EfS.  commitment  to  the  host  nation. 

Further,  the  OHDACA  policy  guidance  specifies  even  higher-order  strategic  objectives: 

•  Increase  military  access  to  the  community  and  country. 

•  Improve  military  influence  in  the  community. 

•  Increase  the  legitimacy  of  local  officials  in  the  eyes  of  the  community. 

•  Create  a  better  public  image  of  Americans,  particularly  the  U.S.  military. 

OHDACA  2008  policy  guidance  stipulates  that  HA  projects  must  address  all  these 
higher-order  strategic  objectives  and  be  consistent  with  overall  security  cooperation  guidance: 
HA  activities  must  contribute,  at  least  in  some  small  way,  to  the  achievement  of  the  coun¬ 
try  campaign  plan,  regional  campaign  plan,  theater  campaign  plan,  and  Guidance  for  the 
Employment  of  the  Force  (GEF).  Project  nominations  must  be  explicit  on  these  points. 

Determining  the  full  range  of  results  that  the  project  is  trying  to  achieve  is  the  basis  for 
identifying  appropriate  MOEs.  MOEs  (and  MOPs,  for  that  matter)  should  also  be  linked,  to 
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the  greatest  extent  possible,  to  the  objectives  that  HA  projects  are  required  to  address.  Working 
through  those  connections  while  a  project  is  being  developed  makes  it  easier  to  track  progress 
and  communicate  success. 

The  next  section  describes  a  tool,  called  an  “objective  tree,”  to  develop  MOEs  by  framing 
project  activities  as  the  “causes”  to  help  produce  desired  “effects.”  The  approach  itself  provides 
a  way  to  think  through  what  needs  to  be  done  to  achieve  the  project’s  objectives — including 
required  inputs,  activities,  outputs,  and  outcomes — and  then  identify  the  indicators  that  will 
best  communicate  progress  and  outcomes  to  others. 

Tool  to  Analyze  Causes  and  Effects 

An  HA  project,  such  as  building  a  village  well,  can  be  understood  within  many  contexts, 
including  its  social  and  strategic  contexts,  its  relation  to  policy  guidance,  and  the  ways  in 
which  it  might  fulfill  a  range  of  tactical  and  strategic  objectives.  How  can  MOEs  be  identified 
in  a  way  that  acknowledges  this  complexity  while  ensuring  that  assessment  remains  systematic 
and  relatively  simple  to  carry  out? 

The  objective  tree  (see  Figure  2.1)  is  a  tool  that  guides  project  planners  and  managers  in 
determining  MOEs  by  displaying  causes  and  effects  in  a  systematic  and  logical  way.  The  objec¬ 
tive  tree  is  a  well-tested  analytic  tool  that  planners  have  used  for  more  than  40  years  to  display 
cause-effect  relationships.1 

Each  box  in  the  objective  tree  represents  a  cause  that  is  necessary  to  achieve  the  effect 
above  it.  The  boxes  represent  completed  actions  (e.g.,  “well  is  built”)  rather  than  processes  (e.g., 
“build  well”).  Each  level  in  the  objective  tree  identifies  the  necessary  and  sufficient  actions 
needed  to  move  up  to  the  next  level.  For  example,  if  a  project  accomplishes  A,  B,  and  C,  it  will 
accomplish  F,  and  if  the  project  accomplishes  D  and  E,  it  will  accomplish  G.  Together,  F  and 
G  will  result  in  the  accomplishment  of  H.  Because  this  is  an  objective  tree,  completed  actions 
are  considered  “objectives.”  This  keeps  the  focus  on  accomplishments,  rather  than  on  the  pro¬ 
cesses  or  activities  undertaken  to  reach  the  objectives. 

Figure  2.1 

Objective  Tree  Tool 


Effect 

A 


Why? 


How?  - ►  What 

else? 


T 

Cause 


NOTE:  Each  level  identifies  the  necessary  and  sufficient  factors  to  achieve  the  level  above. 
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1  See,  for  example,  D.  McLean,  The  Logical  Framework  in  Research  Planning  and  Evaluation ,  Washington,  D.C.:  Interna¬ 
tional  Service  for  National  Agriculture  Research,  Working  Paper  No.  12,  June  1988. 
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The  arrows  in  the  objective  tree  itself  always  go  up  to  show  the  direction  of  cause  and 
effect.  However,  the  cause-effect  relationships  displayed  in  the  tool  can  be  examined  from  all 
directions.  Moving  up  through  the  tree  should  answer  why  the  project  intends  to  undertake 
the  activities  below.  Moving  down  through  the  tree  should  explain  how  the  project  will  achieve 
the  objectives  in  the  levels  above.  And  moving  laterally  through  any  particular  level  explains 
what  else  is  required  (i.e.,  what  other  objectives  must  be  accomplished)  to  achieve  the  objective 
in  the  level  above. 

How  the  objective  tree  tool  works  in  practice  can  be  illustrated  through  a  simple  well¬ 
building  example.  Ideally,  a  project  planner  would  start  with  a  stated  objective  and  work  down 
the  objective  tree  tool  to  identify  the  specific  project  design  and  needs  for  accomplishing  that 
objective.  However,  this  is  not  always  feasible  for  HA  project  planners,  particularly  when  they 
have  entered  the  project  planning  or  execution  process  midstream.  Alternatively,  then,  sup¬ 
pose  that  the  starting  point  for  a  planner  is  a  decision  by  the  planner’s  predecessor  that  a  well 
will  be  built.  The  planner  might  first  ask  logistical  questions  about  how  the  well  will  be  built. 
Figure  2.2  provides  possible  responses:  To  build  the  well,  the  location  must  be  identified,  mate¬ 
rials  acquired,  and  a  construction  team  organized. 

Next,  the  planner  might  ask  what  higher-level  effects  the  project  is  trying  to  achieve  to 
understand  the  project  in  its  context  and  to  determine  how  it  helps  achieve  the  objectives  in  the 
policy  guidance.  An  example  is  shown  in  Figure  2.3.  The  well  is  to  be  built  because  the  project 

Figure  2.2 

Initial  Causes  and  Effects  for  the  Well-Building  Example 
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Figure  2.3 

Fligher-Level  Effects  for  the  Well-Building  Example 
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Importance  of  "Necessary  and  Sufficient" 

An  evaluation  team  traveled  1,500  kilometers  of  roads  built  by  USAID  in  impoverished  western  Kenya  in 
the  late  1970s.  Everyone  the  team  talked  to  said  that  they  liked  the  roads  because  if  someone  got  sick,  the 
person  could  be  taken  to  the  hospital  more  easily.  However,  after  dozens  of  interviews,  the  team  never 
found  an  instance  of  a  person  actually  being  taken  to  the  hospital  by  vehicle.  Why?  Almost  everyone  was 
too  poor  to  own  a  vehicle. 

The  same  team  passed  a  woman  carrying  her  bananas  to  market  on  her  head  just  as  a  bus  passed  her  by.  She 
explained  to  the  team  that  the  price  of  the  round-trip  bus  fare  was  the  same  as  what  she  would  get  for  her 
one  stem  of  bananas,  so  she  had  to  walk. 

The  roads  were  necessary,  but  without  an  income-generating  activity,  they  were  not  sufficient  to  increase 
traffic,  much  less  provide  such  benefits  as  getting  people  to  the  hospital  more  quickly. 


planner  or  the  planner’s  predecessor  wants  to  ensure  “sustained  access  to  clean  water”  in  the 
community.  The  causes  in  the  bottom  row  of  the  figure  lead  to  the  first  effect  (“well  built”), 
which,  in  turn,  becomes  a  cause  of  the  higher-level  effect  of  “sustained  access  to  clean  water.” 

The  tree  is  particularly  useful  for  determining  sufficient  causes  to  produce  desired  results. 
The  items  included  in  all  the  boxes  at  a  given  level  must  collectively  accomplish  all  that  is  nec¬ 
essary  and  sufficient  to  achieve  the  results  shown  in  the  next  level  above. 

The  planner  keeps  asking,  “What  else  is  necessary?”  to  produce  the  desired  results  until 
all  of  the  causes  that  are  sufficient  to  produce  the  next-level  result  are  listed.  Years  of  evalua¬ 
tions  in  the  civilian  world  have  shown  that  a  failure  to  address  all  the  factors  needed  to  achieve 
a  desired  result  (i.e.,  not  asking,  “What  else?”;  forgetting  something  important) — and  thus 
not  meeting  the  “necessary-and-sufficient”  criteria — is  perhaps  the  most  common  reason  that 
projects  fall  short  of  expectations.  Continuing  with  the  example  of  the  village  well  and  looking 
further  at  the  desired  result  of  sustained  access  to  clean  water,  the  planner  asks,  “What  else, 
beyond  building  the  well,  is  necessary  for  sustained  access  to  clean  water?”  Because  something 
could  happen  to  jeopardize  the  well  or  its  water  quality,  the  planner  determines  that  the  well 
must  be  maintained  in  a  way  that  will  keep  the  water  “clean”  and  that  there  must  be  a  way  to 
ensure  this.  In  this  case,  the  planner  is  working  downward  from  the  broader  objective  of  access 
to  clean  water,  consistent  with  the  logical  flow  from  objectives  to  inputs  in  planning  a  project. 

Figure  2.4  thus  extends  the  objective  tree  to  incorporate  the  idea  that  workers  must  be 
trained  to  maintain  the  well  (“Tocals  trained  to  maintain  the  well,”  in  the  objective  tree),  a 
second  necessary  cause  to  produce  the  effect  of  sustained  access  to  clean  water.  The  next  ques¬ 
tion  to  ask  is,  “Are  these  causes  sufficient  to  produce  the  planned  result?”  If  so,  the  planner 
moves  up  to  the  next  level  of  the  tree;  but  if  not,  the  planner  continues  to  identify  other  causes 
that  are  necessary  to  produce  the  intended  result. 


"Why?"  Matters 

In  Afghanistan,  a  captain  was  responsible  for  distributing  U.S.  food  aid  to  the  provinces  shortly  after  the 
action  began.  The  most  efficient  way  to  distribute  the  food  was  to  put  it  on  the  warlords'  trucks.  The 
captain's  objective  was  accomplished  successfully. 

A  year  later,  when  USG  civilians  were  supporting  local  elections,  most  Afghans  voted  for  the  warlords  who 
had  brought  them  food.  They  might  have  voted  for  the  warlords  anyway  because  they  were  powerful,  but 
what  seemed  like  an  efficient  means  of  food  distribution  also  reinforced  the  power  and  legitimacy  of  one 
group  of  actors  in  the  region. 

Knowing  that  the  higher-order  objective  was  increasing  the  host  nation's  legitimacy  and  influence,  the 
captain  might  have  made  very  different  choices  about  how  food  should  have  been  transported. 
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Figure  2.4 

Sufficient  Activities  to  Achieve  Desired  Effects  in  the  Well-Building  Example 
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Having  now  completed  three  levels  of  the  objective  tree  tool,  the  planner  considers  the 
highest  level  and  next  asks  why  sustained  access  to  clean  water  is  important.  One  answer  that 
ties  into  policy  guidance  and  is  important  in  almost  all  HA  work  is  increasing  the  legitimacy 
of  local  leaders  (see  Figure  2.5).  Working  together  with  local  leaders  to  build  a  well  can  improve 
the  standing  of  the  local  leaders  in  the  eyes  of  their  citizens.  Of  course,  this  is  just  one  example 
among  many.  Another  reason  to  provide  sustained  access  to  clean  water  is  to  improve  health 
(e.g.,  by  reducing  diarrheal  diseases).  It  is  important  to  respond  to  what  matters  in  the  local 
context  and  to  create  an  objective  tree  that  is  responsive  to  those  circumstances  and  connects 
appropriately  to  HA  policy  guidance  and  other  objectives  associated  with  the  country,  theater, 
and  so  on. 


Figure  2.5 

Connecting  to  Fligher-Level  Objectives  in  the  Well-Building  Example 
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Cause-and-Effect  Relationships  in  the  Context  of  the  Project  Cycle 

Now  that  the  relatively  abstract  process  of  linking  objectives  to  project  design  and  activities  has 
been  completed,  it  is  important  to  return  to  the  matter  of  practical  project  planning  and  place 
the  cause-and-effect  relationships  in  the  context  of  the  project  cycle  shown  in  the  top  half  of 
Figure  2.6.  (Note  that  this  figure  is  identical  to  Figure  1.2  in  Chapter  One.) 

The  project  cycle  reflects  both  the  time  dimension  and  the  causal  chain.  That  is,  the 
movement  from  left  to  right  represents  the  passage  of  time  as  well  as  steps  in  the  project’s 
implementation.  In  Figure  2.7,  the  rows  (or  levels)  in  the  objective  tree  created  from  the  well 
example  have  been  linked  with  the  corresponding  components  of  the  project  cycle.  The  shapes 
and  colors  of  the  boxes  follow  those  used  in  the  project  cycle  in  Figure  2.6. 

In  the  example  of  the  village  well,  inputs  for  building  a  well  are  land,  materials,  and 
human  resources.  Those,  in  turn,  lead  to  specific  processes  (or  activities)  to  be  carried  out: 
identifying  a  location,  acquiring  materials,  and  organizing  the  construction  team.  Those  pro¬ 
cesses  will  produce  desired  outputs  (the  well  is  built,  locals  are  trained  to  maintain  the  well), 
which  will  lead  to  the  planned  outcome:  sustained  access  to  clean  water. 

The  objective  tree  demonstration  in  Figure  2.7  includes  a  level  called  “impact,”  which 
is  not  represented  in  the  project  cycle  because  FIA  projects  are  generally  too  small  and  too 
short  to  have  long-term  or  large-scale  impacts  of  their  own.  In  other  words,  building  one  well 
is  unlikely  to  have  a  significant  or  measurable  effect  on  the  legitimacy  of  local  leaders — the 
anticipated  impact  in  this  example.  However,  a  single  HA  project  can  potentially  contribute  to 
improved  public  perceptions  of  local  leaders.  Collectively,  a  group  of  HA  projects  is  more  likely 
to  contribute  to  improved  public  perception  and  thus  enhance  the  legitimacy  of  local  leaders. 

Figure  2.6 

Measuring  Performance  and  Effectiveness:  Links  to  the  Project  Cycle 
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Figure  2.7 

Connecting  the  Objective  Tree  with  the  Project  Cycle 
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Therefore,  impact  indicators  reflecting  higher-order  strategic  objectives  should  be  tracked,  to 
the  extent  possible,  so  that  there  are  data  available  for  larger-scale  assessments  of  multiple  proj¬ 
ects,  country  plans,  theater  campaign  plans,  or  the  OHDACA  program  as  a  whole.  Such  large- 
scale  assessments  (or  “evaluations,”  as  they  are  called  in  the  civilian  world)  are  the  responsibility 
of  other  groups  within  DoD  (e.g.,  assessment  offices  or  technical  assistance  groups)  and  are 
beyond  the  scope  of  this  handbook. 

What  Are  the  MOEs? 

As  discussed  in  Chapter  One,  MOEs  are  a  type  of  indicator  that  directly  measures  outcomes. 
In  the  example  of  building  a  well,  the  desired  outcome  to  be  measured  is  that  the  community 
around  the  well  has  sustained  access  to  clean  water.  How  does  one  measure  access  to  clean 
water?  This  outcome  or  objective  has  two  initial  components  that  need  to  be  captured:  access 
must  be  improved  and  the  water  must  be  clean.  When  indicators  are  measured  over  time,  they 
capture  a  third  component:  sustainability  of  the  desired  outcomes  of  access  and  clean  water. 

•  Sustained  access :  At  its  most  basic,  this  means  that  the  local  community  can  get  water  out 
of  the  well.  The  well  needs  to  be  sufficiently  close  to  the  community,  and  it  must  produce 
enough  water  at  a  reasonable  rate  to  meet  the  needs  of  the  community.  Traditionally, 
the  World  Health  Organization  defines  “access”  to  drinking  water  in  terms  of  distance, 
time,  and  flow  from  the  well.  For  example,  the  proportion  of  the  population  that  lives 
within  500  meters  (or  10  minutes)  walking  time  to  the  well  has  “access”  to  the  well.  The 
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well  should  provide  water  at  a  rate  that  minimizes  waiting  time  (15-30  minutes)  and  that 
ensures  that  the  fill  time  for  a  water  container  is  short  (no  more  than  three  minutes  to  fill 
a  20-liter  container).  Each  is  a  possible  indicator  of  access. 

•  Clean  water:  The  determination  of  whether  water  is  clean  technically  requires  some  kind 
of  testing  of  water  quality.  Generally,  such  a  standard  would  translate  into  an  indica¬ 
tor,  such  as  “number  of  fecal  coliforms  per  100  milliliters  at  the  point  of  distribution.” 
Although  it  is  required  for  DoD  HA  well-building  projects,  in  some  cases,  the  chemical 
test  to  accurately  measure  coliform  levels  might  be  infeasible.  Thus,  the  “test”  (and  related 
indicator)  might  instead  consist  of  a  visual  inspection  to  determine  that  nothing  obvious 
has  contaminated  the  well  and  that  the  water  in  the  well  looks  satisfactory  (not  turbid 
or  cloudy).  Both  are  MOEs  that  measure  clean  water  at  two  different  levels  of  precision. 
Which  one  is  used  is  up  to  the  local  project  planner,  but  in  general,  larger-scale  or  more 
strategically  important  projects  will  require  the  more  precise  indicator  of  water  quality. 

The  indicators  for  “sustained  access”  and  “clean  water”  are  first  collected  as  baseline  mea¬ 
sures  and  then  as  immediate  outcome  indicators,  reflecting  the  success  of  the  project  at  its 
completion.  The  same  indicators  would  be  measured  again  a  year  after  project  completion, 
allowing  an  evaluation  of  the  “sustained”  part  of  the  project  objective. 

As  explained  in  Chapter  One,  in  most  cases,  the  same  outcome  indicators  should  be 
collected  at  baseline  (before  the  project),  immediately  after  project  completion,  and  during 
follow-up  after  one  year — that  is,  data  should  be  collected  using  the  same  indicators  in  differ¬ 
ent  periods  for  purposes  of  direct  comparison  over  time.  This  is  the  best  way  to  communicate 
what  a  project  really  accomplished.  Using  different  data  (say,  from  another  village)  or  different 
indicators  (associated  with  water  disinfectants,  for  example)  at  these  different  times  will  not 
allow  the  comparisons  needed  to  document  what  the  project  really  achieved. 

It  is  clear  that  even  a  simple  objective,  such  as  sustained  access  to  clean  water,  might  have 
more  than  one  possible  MOE.  This  chapter  provided  just  a  few  possible  MOEs  for  the  pri¬ 
mary  objective  in  this  example  project.  Developing  MOEs  can  be  daunting.  To  make  it  easier, 
Chapter  Three  describes  indicators  in  more  detail  and  provides  a  set  of  indicators  for  use  across 
all  HA  projects,  along  with  indicators  for  each  common  type  of  HA  project  (water  and  sanita¬ 
tion,  construction  of  health  infrastructure,  construction  of  nonhealth  infrastructure,  health 
services,  and  disaster-related  projects).  In  fact,  most  of  the  example  MOEs  in  the  well-building 
example  were  selected  from  the  indicators  in  Chapter  Three  and  the  Step-by-Step  User’s  Guide 
for  Project  Assessment  (Part  II  of  this  handbook).  Given  the  MOEs  provided  in  Chapter  Three, 
additional  MOE  development  will  be  necessary  only  for  HA  projects  with  unique  objectives  or 
those  that  do  not  fall  into  one  of  the  major  preidentified  project  types. 

It  should  be  noted  that,  sometimes,  it  is  impossible  to  gather  direct  indicators  for  a  proj¬ 
ect  outcome  because,  for  example,  the  outcome  is  abstract  or  it  is  too  difficult  or  expensive  to 
measure  directly.  In  those  cases,  it  may  be  necessary  to  identify  appropriate  proxy  indicators 
associated  with  an  outcome.  A  proxy  is  not  a  direct  measure  or  indicator,  but  it  is  either  highly 
suggestive  or  highly  correlated  with  what  the  project  is  trying  to  achieve.  More  details  regard¬ 
ing  proxy  indicators,  including  some  examples,  are  provided  at  the  end  of  Chapter  Three. 
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How  to  Determine  Measures  of  Performance 

To  know  whether  the  project  is  making  progress  toward  the  achievement  of  its  planned  results, 
managers  need  to  follow  the  execution  of  the  project.  Just  as  data  for  the  indicators  of  immedi¬ 
ate  and  sustained  outcomes  (MOEs)  must  be  collected,  so  too  must  data  for  those  indicators 
that  directly  measure  the  progress  of  the  project — the  inputs,  processes  (activities)  and  outputs 
(MOPs).  Figure  2.8  returns  to  the  well-building  example  but  shows  indicators  in  the  shadow 
boxes  below  the  objectives.  For  simplicity,  only  one  indicator  is  shown  below  each  box,  but 
there  may  be  more  than  one  indicator  for  each,  depending  on  the  scope  and  complexity  of  the 
project. 

What  Are  the  MOPs? 

In  Figure  2.8,  the  MOP  for  choosing  the  location  could  be  a  simple  “yes”  or  “no”  question 
answered  by  the  project  manager.  Similar  MOPs  would  be  chosen  for  the  acquisition  of 
materials,  organizing  the  construction  team,  or  contracting  issues  that  may  be  associated  with 
the  well  project.  With  respect  to  outputs,  MOPs  may  be  specific  (e.g.,  well  built  to  relevant 
local,  international,  or  national  standards,  as  appropriate),  or  they  may  reflect  a  general  output 
(e.g.,  well  built  in  a  way  that  enables  local  maintenance  over  time).  Again,  there  may  be  mul¬ 
tiple  MOPs  for  a  given  input,  process,  or  output. 

Figure  2.8 

MOPs  and  MOEs  Associated  with  the  Objective  Tree  in  the  Well-Building  Example 
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Objectives  Drive  Project  Design,  MOEs,  and  MOPs 

Figure  2.8  includes  an  indicator  for  the  impact  of  “enhanced  legitimacy  of  local  leaders.”  This 
is  beyond  the  FIA  project  cycle,  but  the  project  planner  or  other  country  team  member  may 
want  to  consider  collecting  project-specific  data  of  this  type  for  two  reasons: 

•  Knowing  and  understanding  the  higher-order  objective  will  affect  how  the  project  is 
implemented  and,  just  as  importantly,  why  and  how  achievements  should  be  measured  as 
they  relate  to  higher-order  (strategic)  objectives. 

•  OFIDACA  program  managers  will  be  evaluating  selected  projects  (or  groups  of  projects). 
They  will  need  data  on  individual  projects  as  they  assess  the  collective  achievements  of  a 
larger  number  of  projects. 

In  this  case,  a  focus  on  the  higher-order  objective  of  enhancing  the  legitimacy  of  local 
leaders  would  cause  project  planners  to  carry  out  the  project  in  a  different  way.  For  example,  it 
would  be  most  efficient  to  simply  have  a  local  official  designate  a  site  for  the  well.  But  if  both 
the  local  community  and  the  local  leaders  are  involved  in  site  selection  or  the  ribbon-cutting 
ceremony,  those  leaders  will  be  seen  as  part  of  the  action  of  bringing  the  well  to  the  com¬ 
munity,  which  should  enhance  their  legitimacy  with  local  citizens.  This  consideration  would 
modify  the  project  plan  and  the  causal  chain,  as  shown  in  Figure  2.9. 

With  this  higher-order  objective  in  mind,  the  activity  or  process  of  “location  identified” 
in  Figure  2.8  is  now  “location  identified  in  cooperation  with  local  community  leaders”  in 
Figure  2.9;  similarly,  “construction  team  organized”  is  now  “construction  organized  by  local 
government.”  A  change  in  the  causes  at  the  bottom  of  the  objective  tree  would,  in  turn,  change 


Figure  2.9 

Harmonizing  the  Objective  Tree 
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some  of  the  indicators  at  each  level.  For  example,  the  simple  yes/no  indicator  was  hne  for  docu¬ 
menting  that  the  location  was  identified,  but  this  is  no  longer  the  point.  How  might  the  indi¬ 
cator  be  changed  to  better  reflect  the  importance  of  involving  local  leaders  so  that  the  project 
better  responds  to  the  higher-order  “legitimacy”  objective?  In  addition,  the  inputs  required 
to  execute  the  newly  stated  activities  or  processes  will  expand.  Land  will  still  be  required  for 
“location  identified  in  cooperation  with  local  leaders,”  but  “meetings  with  local  leaders”  will 
also  be  a  required  input.  Similarly,  “construction  organized  by  local  government”  will  require 
not  only  materials  but  also  a  local  government  committee. 

What  is  even  more  important  than  the  names  of  the  primary  indicators  is  an  understand¬ 
ing  that  activities  or  processes  (e.g.,  identifying  the  location,  acquiring  materials,  organizing 
the  construction  team)  should  lead  to  outputs  (e.g.,  well  is  built),  which  lead  to  outcomes  (e.g., 
sustained  access  to  clean  water) — and  that  a  manager  must  document  whether  or  not  this  has 
occurred.  Knowing  which  objectives  have  been  achieved  and  where  problems  remain  allows  a 
manager  to  take  corrective  actions  to  ensure  a  project  succeeds  in  meeting  its  direct  objectives 
and,  in  turn,  successfully  contributes  to  higher-level  objectives. 


CHAPTER  THREE 


Indicators  for  Humanitarian  Assistance  Projects 


This  chapter  presents  indicators  that  can  be  used  as  measures  of  performance  (MOPs)  and 
measures  of  effectiveness  (MOEs)  for  OHDACA-funded  HA  projects.  They  may  also  be  useful 
for  other  DoD  HA  projects  (for  example,  humanitarian  and  civic  assistance  activities).  The 
indicators  are  divided  into  three  major  categories:  sets  of  required  project  management  indi¬ 
cators,  core  indicators  to  be  collected  across  all  HA  projects,  and  indicators  related  to  spe¬ 
cific  project  types,  as  required  for  each  main  type  of  HA  project:  water  and  sanitation,  health 
infrastructure,  nonhealth  infrastructure,  health  services,  and  disaster-related  projects.  Stan¬ 
dardizing  indicators  facilitates  comparisons  across  HA  projects  and  aggregation  of  data  for 
higher-level  assessment  at  the  country,  COCOM  or  theater,  and  DoD  (e.g.,  OHDACA  or 
security  cooperation  program)  levels.  To  allow  clear  comparisons,  project  managers  need  to 
make  measurements  related  to  intended  outcomes  both  at  project  startup  (baseline  data  before 
a  project  begins)  and  after  a  project  is  complete  (immediately  after  project  completion  and 
again  after  one  year). 

The  standard  indicators  presented  in  this  chapter  are  required,  but  they  do  not  necessarily 
represent  all  that  are  needed.  Project  planners  may  need  to  choose  existing  indicators  used  by 
other  entities,  such  as  USAID,  or  develop  additional  MOE  and  MOP  indicators  to  measure 
the  achievement  of  their  project-specific  objectives  using  the  guidance  provided  in  Chapter 
Two  and  at  the  end  of  this  chapter. 

The  first  section  of  this  chapter  reviews  how  the  required  indicators  were  developed.  The 
second  section  provides  a  review  and  explanation  of  the  required  core  indicators,  and  the  third 
section  provides  a  review  and  explanation  of  the  indicators  for  five  types  of  HA  projects.  The 
final  sections  of  the  chapter  provide  guidance  on  choosing  (or  developing)  additional  project- 
specific  indicators  when  a  project  has  unique  objectives  that  cannot  be  measured  by  the  core 
indicators  or  when  a  project  does  not  fall  into  one  of  the  five  project  categories  presented.  We 
conclude  this  chapter  with  additional  tips  regarding  proxy  indicators  and  avoiding  bias  in 
designing  indicators. 


How  the  Indicators  Were  Developed 

The  indicators  presented  here  are  required  for  DoD  OHDACA-funded  projects.  They  are 
intended  to  provide  a  well-balanced  picture  of  HA  project  activities  and  their  results.  The 
development  of  these  indicators  involved  consideration  of  HA  and  security  cooperation  guid¬ 
ance,  the  GEF,  the  DoD  HA  project  portfolio,  and  other  information  regarding  the  future 
directions  of  HA  activities.  As  described  in  the  Step-by-Step  User’s  Guide  for  Project  Assess- 
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ment  (Part  II  of  this  handbook),  the  project  management  indicators  reflect  various  aspects 
of  performance  and  collaboration  with  other  USG  agencies,  since  national  and  military  policy 
require  interagency  cooperation  (e.g.,  in  stability  operations).  The  core  indicators  reflect 
higher-order  DoD  strategic  objectives  to  which  all  DoD  HA  activities  should  be  contribut¬ 
ing  to  some  extent.  The  indicators  related  to  specific  project  types  reflect  typical  project 
objectives  for  the  most  common  OHDACA-funded  HA  projects.  Often,  project  planners  will 
supplement  these  indicators  with  those  that  reflect  their  own  unique  project  objectives. 

The  required  indicators  can  be  collected  easily  by  using  the  worksheets  at  the  end  of  the 
Step-by-Step  User’s  Guide  for  Project  Assessment  and  on  the  CD  that  accompanies  this  hand¬ 
book.  The  worksheets  can  be  used  to  manage  projects,  extract  lessons  learned,  and  prepare 
after-action  reports.  Input  into  the  OHASIS  database  will  facilitate  data  storage,  generation 
of  after- action  reports,  handing  projects  over  to  subsequent  DoD  or  civilian  personnel  to  com¬ 
plete  the  assessment  process,  and  the  ability  to  “roll  up”  information  across  many  projects  to 
inform  broader-level  assessments.  The  indicators  combine  qualitative  and  quantitative  infor¬ 
mation  to  provide  a  complete  picture  of  the  project. 

The  indicators  are  largely  drawn  from  established  international  humanitarian  assistance 
and  development  sources,  such  as  the  Sphere  Project  (a  coalition  of  NGOs  involved  with  disas¬ 
ter  response  and  humanitarian  assistance  that  have  created  a  set  of  disaster-response  standards 
and  related  indicators),  the  United  Nations  Development  Programme  and  other  UN  offices, 
USAID,  the  Department  of  State,  and  other  organizations.  These  indicators  have  been  devel¬ 
oped  and  tested  in  applicable  environments  for  ease  of  collection,  reliability  (consistency),  and 
validity  (accuracy).  Using  previously  tested  indicators  makes  communicating  across  projects 
and  across  agencies  much  easier. 


Management  and  Core  Indicators  Required  for  All  HA  Projects 

Indicators  required  for  all  HA  projects  are  presented  in  Tables  3.1  and  3.2.  Table  3.1  presents 
input  and  process  indicators  that  are  related  to  project  management.  Table  3.2  presents  core 
indicators  for  all  HA  projects  and  describes  how  they  map  to  higher-order  strategic  objectives. 
In  that  table,  the  first  column  lists  the  highest-order  strategic  objective  to  which  the  indicators 
contribute,  the  second  column  shows  the  project-specific  objective,  and  the  third  column  lists 
the  indicators  themselves.  The  fourth  column  identifies  how  the  indicator  fits  into  the  project 
cycle  (i.e.,  input,  process,  output,  or  outcome),  the  fifth  column  defines  the  indicator  as  an 
MOP  or  MOE,  and  the  final  column  provides  at  least  one  method  of  collecting  data  for  each 
indicator. 

Overall,  the  indicators  are  intended  to  be  straightforward  in  terms  of  what  should  be 
measured.  It  is  important  to  remember,  however,  that  there  are  resources  that  can  and  should 
be  drawn  upon  for  assistance  and  support  in  collecting  indicator  data.  For  example,  USAID 
colleagues  in  country  are  likely  to  have  experience  in  similar  data  collection  exercises  and  can 
be  of  assistance  in  the  areas  of  methodology  and  logistics.  University  students  and  faculty,  as 
well  as  other  host-nation  counterparts,  are  also  likely  to  be  valuable  resources  and  should  be 
tapped  as  appropriate.  Worksheets  for  collecting  core  indicator  data  can  be  found  in  the  user’s 
guide. 
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Table  3.1 

Project  Management  Indicators  for  All  Projects 


Aspect  of  Project 

Indicator 

Indicator 

Indicator 

Data 

Management 

Type 

Family 

Source 

Project  planning 

Project  assessment  plan  developed  before  project 
execution  (Y/N) 

Input 

MOP 

Self-report 

Post-project  sustainment  plan  developed  (Y/N) 

Input 

MOP 

Self-report 

Project  timeline 

Project  completed  by  original  deadline  (Y/N) 

(if  no,  explain  briefly) 

Process 

MOP 

Self-report 

Project  budget 

Project  completed  within  original  budget  (Y/N) 

(if  no,  explain  briefly) 

Process 

MOP 

Self-report 

Coordination 

Project  reviewed  and  discussed  with  relevant  State 

across  USG 

Department  officials  and  feedback  incorporated 

Input 

MOP 

Self-report 

agencies 

(Y/N/NA) 

Project  reviewed  and  discussed  with  relevant  USAID 
officials  (including  Office  of  Foreign  Disaster  Assistance) 
and  feedback  incorporated  (Y/N/NA) 

Input 

MOP 

Self-report 

Project  reviewed  and  discussed  with  other  relevant  USG 
officials  (specify)  and  feedback  incorporated  (Y/N/NA) 

Input 

MOP 

Self-report 

Project  reviewed  and  discussed  with  other  relevant 
agencies  (indicate  which  type  and  specify  which  agency) 
and  feedback  incorporated  (Y/N/NA) 

•  International  organizations 

•  Other  donor  nations 

Input 

MOP 

Self-report 

•  Local  and  international  NGOs 

•  Relevant  private-sector  representatives 

•  Third-party  allied/coalition  militaries 

USAID  provided  financial,  human,  or  in-kind  resources 
(Y/N/NA) 

Input 

MOP 

Self-report 

Other  USG  counterpart  (specify)  provided  financial, 
human,  or  in-kind  resources  (Y/N/NA) 

Input 

MOP 

Self-report 

Host  nation  agreed  to  sustain  the  project 

Input 

MOP 

Self-report 

Project 

Problems  impeded  project  execution  (Y/N)  (if  yes,  specify) 

Process 

MOP 

Self-report 

troubleshooting 

Problems  resolved  satisfactorily  (Y/N)  (if  yes,  explain  how; 
if  no,  explain  why) 

Process 

MOP 

Self-report 

Project 

Project  assessed  with  reference  to  stated  objectives  (Y/N) 

Process 

MOP 

Self-report 

assessment 

Baseline  assessment  completed  (Y/N,  date  completed) 

Process 

MOP 

Self-report 

Baseline  assessment  conducted  with  at  least  one 
community  member  (e.g.,  leader,  supervisor)  (Y/N/NA) 

Process 

MOP 

Self-report 

Immediate  post-project  assessment  completed  (Y/N, 
date  completed) 

Process 

MOP 

Self-report 

Immediate  post-project  assessment  conducted  with  at 
least  one  community  member  (Y/N/NA) 

Process 

MOP 

Self-report 

One-year  post-project  assessment  completed  (Y/N, 
date  completed) 

Process 

MOP 

Self-report 

One-year  post-project  assessment  conducted  with  at 
least  one  community  member  (Y/N/NA) 

Process 

MOP 

Self-report 

Additional  post-project  assessment  completed 
(date/years  post-completion) 

Process 

MOP 

Self-report 

Table  3.2 

Core  Indicators  for  All  Projects,  Related  to  Strategic  Objectives 


Project-Specific 

Indicator 

Indicator 

Data 

Strategic  Objective 

Objective 

Indicator 

Type 

Family 

Source 

Capacity-building 

Capacity  of  host- 
nation  civilians 

Project  targets  transfer  of  knowledge  or  skills  from  DoD  to  host-nation  civilians 

Input 

MOP 

Self-report 

increased 

Project  implemented  in  collaboration  with  host-nation  civilian(s)  (Y/N) 

Process 

MOP 

Self-report 

(if  applicable) 

Number  of  host-nation  civilians  trained  during  project  execution 

Output 

MOP 

Self-report 

Host-nation  civilians  operate  and  maintain  project/property  once  project  is 
completed  (Likert  scale  0-4) 

Outcome 

MOE 

Self-report 

Capacity  of  host- 

Project  targets  transfer  of  knowledge  or  skills  from  DoD  to  host-nation  government 

Input 

MOP 

Self-report 

nation  government 
(nonmilitary) 

Project  implemented  in  collaboration  with  host-nation  government  (nonmilitary) 

Process 

MOP 

Self-report 

increased 

(Y/N)  (specify) 

(if  applicable) 

Number  of  host-nation  government  (nonmilitary)  personnel  trained  during  project 
execution 

Output 

MOP 

Self-report 

Host-nation  government  (nonmilitary)  operates  and  maintains  project/property 
once  project  is  completed  (Likert  scale  0-4) 

Outcome 

MOE 

Self-report 

Capacity  of  host- 
nation  military 

Project  targets  transfer  of  knowledge  or  skills  from  DoD  to  host-nation  military 

Input 

MOP 

Self-report 

increased  for 
civilian  benefit 

Project  implemented  in  collaboration  with  host-nation  military  (Y/N)  (specify) 

Process 

MOP 

Self-report 

(if  applicable) 

Number  of  host-nation  military  personnel  trained  during  project  execution 

Output 

MOP 

Self-report 

Host-nation  military  operates  and  maintains  project/property  once  project  is 
completed  (Likert  scale  0-4) 

Outcome 

MOE 

Self-report 

Positive  visibility 

Project  recognized 

Project  sends  a  tangible  signal  in  the  host  nation,  regionally,  and  globally  that  DoD 

Input 

MOP 

Self-report 

as  DoD/USG  effort 

and  USG  respond  to  humanitarian  needs  and  have  an  interest  in  the  well-being  of 

and  received 

those  in  need 

positively  by  the 
community 

Project  information  recognizing  DoD  and  host  nation  is  disseminated  in  local 
community  (Y/N) 

Process 

MOP 

Self-report 

DoD  or  other  USG  representative  present  at  the  ribbon-cutting  or  other  ceremony 
(Y/N/NA)  (specify) 

Output 

MOP 

Self-report 

Project  includes,  where  appropriate,  some  tangible,  visible,  or  substantive  marker 

Output 

MOE 

Self-report 

(e.g.,  cornerstone,  plaque,  sign)  of  both  DoD  and  host-nation  involvement,  with 
the  host  nation  in  the  lead  (Y/N/NA)  (describe  marker  or  why  not  applicable/ 
appropriate) 
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Table  3.2 — Continued 


Project-Specific 

Indicator 

Indicator 

Data 

Strategic  Objective 

Objective 

Indicator 

Type 

Family 

Source 

Positive  visibility 

Project  recognized 

Extent  to  which  local  community  feels  project  will  improve  quality  of  life  (Likert 

Outcome 

MOE 

Poll,  local 

(cont.) 

as  DoD/USG  effort 
and  received 

scale  0-4) 

consultation 

positively  by  the 

Extent  to  which  local  community  feels  the  project  was  responsive  to  local  needs 

Outcome 

MOE 

Poll,  local 

community  (cont.) 

(Likert  scale  0-4) 

consultation 

Degree  of  favorable  impression  of  the  United  States  (Likert  scale  0-4) 

Outcome 

MOE 

Poll 

Extent  to  which  the  project  changed  community  opinion  about  the  United  States 

Outcome 

MOE 

Poll,  local 

and  whether  it  was  in  a  positive  or  negative  way 

consultation 

Access/influence 

Cooperated  with 

Project  developed  in  response  to  explicit  host-nation  civilian  need  or  request  (Y/N) 

Input 

MOP 

Self-report 

host-nation 

(specify  organization^]) 

civilians 

(if  applicable) 

Project  designed  in  coordination  with  host-nation  civilians  (Y/N)  (Specify 
organization^]) 

Input 

MOP 

Self-report 

Host-nation  civilians  provided  information  or  financial,  human,  or  in-kind  resources 
for  the  project  (Y/N/NA) 

Input 

MOP 

Self-report 

Cooperated 

Project  developed  in  response  to  explicit  host-nation  (nonmilitary)  need  or  request 

Input 

MOP 

Self-report 

with  host-nation 

(Y/N)  (specify  organization^]) 

government 

(nonmilitary) 

Project  designed  in  coordination  with  host-nation  government  (nonmilitary)  official 

Input 

MOP 

Self-report 

(if  applicable) 

(Y/N)  (specify  organization^]) 

Host  nation  provided  information,  financial,  human,  or  in-kind  resources  for  project 
(Y/N/NA) 

Input 

MOP 

Self-report 

Cooperated  with 

Project  developed  in  response  to  explicit  host-nation  military  need  or  request  (Y/N) 

Input 

MOP 

Self-report 

host-nation  military 
for  civilian  benefit 

(specify  organization^]) 

(if  applicable) 

Project  designed  in  coordination  with  host-nation  military  officials  (Y/N)  (specify 
organization^]) 

Input 

MOP 

Self-report 

Host-nation  military  provided  information  or  financial,  human,  or  in-kind  resources 
(Y/N/NA) 

Input 

MOP 

Self-report 

Project  enhanced 

At  least  one  host-nation  government  official  (military  or  nonmilitary)  present  at  the 

Output 

MOP 

Self-report 

host-nation 

groundbreaking  or  ribbon-cutting  ceremony  (Y/N) 

government's 
legitimacy  among 

Project  outputs  are  operated,  maintained,  or  managed  by  the  host-nation 

Outcome 

MOE 

Self-report 

the  public 

government  (military  or  nonmilitary)  (Y/N/NA) 

NOTE:  The  Likert  scale  assesses  level  of  agreement  with  a  statement.  Here,  0  =  not  at  all,  1  =  small  amount,  2  =  medium  amount,  3  =  large  amount,  and  4  =  completely. 
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Indicators  Related  to  Specific  HA  Project  Types 

Indicators  specific  to  different  types  of  HA  projects  are  presented  in  Tables  3.3  through  3.7, 
using  the  same  format  as  was  used  for  the  core  indicators.  The  five  main  types  of  HA  projects 
currently  undertaken  by  COCOMs  are  described  below,  with  examples  of  each. 

•  Water  and  sanitation  projects:  The  majority  of  water  and  sanitation  projects  focus  on 
building  wells.  Water  treatment  projects  that  usually  have  the  objective  of  increasing  or 
improving  potable  water  also  fall  into  this  category.  These  projects  have  been  proposed 
and  implemented  across  COCOMs.  In  other  development  contexts,  the  construction 
of  toilet  facilities  and  latrines,  solid  waste  disposal  areas,  and  projects  to  improve  com¬ 
munity  drainage  are  also  considered  water  and  sanitation  projects.  Indicators  for  each  of 
these  types  of  projects  are  presented  in  Table  3.3. 

•  Health  infrastructure  projects:  Health  infrastructure  projects  include  the  construction  or 
refurbishing  of  existing  health  care  facilities  (e.g.,  hospitals,  clinics).  These  construction 
projects  have  been  undertaken  by  all  COCOMs.  Indicators  for  this  type  of  project  are 
presented  in  Table  3.4. 

•  Nonhealth  infrastructure  projects:  Nonhealth  infrastructure  projects  include  the  construc¬ 
tion  or  refurbishing  of  schools,  roads,  orphanages,  community  centers,  women’s  centers, 
and  warehouses  or  any  other  type  of  construction  project.  All  these  nonhealth  infrastruc¬ 
ture  projects  have  been  implemented  by  one  or  more  COCOMs.  Indicators  for  this  type 
of  project  are  presented  in  Table  3.5. 

•  Health  services  projects:  Health  services  projects  have  traditionally  been  supported  pri¬ 
marily  by  Humanitarian  and  Civic  Assistance — not  HA  OHDACA — funds.  However, 
more  HA  funds  may  be  provided  for  medical  service  provision  in  the  future,  including 
emergency  or  disaster-related  medical  care  or,  possibly,  direct  clinic  or  hospital  care  in 
high-need  areas  or  training  of  health  personnel.  The  proportion  of  health  services  projects 
is  expected  to  grow  in  the  coming  years.  Indicators  for  this  type  of  project  are  presented 
in  Table  3.6. 

•  Disaster-related  projects:  As  discussed  earlier,  disaster-related  projects  generally  overlap 
with  one  of  the  previously  mentioned  categories  but  are  typically  associated  with  one  or 
more  phases  in  the  disaster  cycle:  preparedness,  response,  or  recovery  and  reconstruc¬ 
tion.  These  projects  may  include  developing  disaster-resistant  shelters  or  other  structures, 
building  warehouses  for  emergency  supplies,  building  drainage  areas  for  flood  control 
in  low-lying  areas,  or  the  provision  of  emergency  medical  care  immediately  following  a 
disaster.  Indicators  for  this  type  of  project  are  presented  in  Table  3.7. 


Table  3.3 

Water  and  Sanitation  Project  Indicators 


Project-Specific 

Indicator 

Indicator 

Strategic  Objective 

Objective 

Indicator 

Type 

Family 

Data  Source 

Humanitarian 

Access  to  improved 

Number  and  location  of  potable  water  sources  created 

Output 

MOP 

Self-report, 

impact 

source  of  drinking 
water  enhanced 

geocoded  data 

(if  applicable) 

Well  built  to  relevant  local  (and  international  or  national,  as  appropriate)  standards 
(Y/N) 

Output 

MOP 

Self  report 

Well  built  in  a  way  that  enables  local  maintenance  over  time  (Likert  scale  0-4) 

Output 

MOP 

Site  inspection 

Number  of  people  obtaining  water  from  source  daily,  divided  by  number  of  people 

Outcome 

MOE 

Site  inspection 

living  within  500  m  or  a  10-minute  walk  of  safe  drinking  water  source 

and/or  household 

survey 

Water  quality 
enhanced 

Drinking  water  from  source(s)  created  by  project  is  nonturbid  by  inspection3  (Y/N) 

Outcome 

MOE 

Site  inspection 

(if  applicable) 

Number  of  fecal  coliforms  per  100  ml  at  distribution  point 

Outcome 

MOE 

Chemical  test 

Number  of  fecal  coliforms  per  100  ml  at  point  of  use 

Outcome 

MOE 

Chemical  test 

Access  to  excreta 
disposal  facilities 

Number  of  latrines  dug  or  other  excreta  disposal  facilities  created 

Output 

MOP 

Self-report 

increased 

Number  of  people  using  excreta  disposal  daily,  divided  by  number  of  people  living 

Outcome 

MOE 

Self-report 

(if  applicable Jb 

within  50  m  of  adequate  excreta  disposal 

Excreta  disposal  facility  created  meets  relevant  local  (and  international  or  national, 
as  appropriate)  standards  of  quality  (Y/N) 

Outcome 

MOE 

Self-report 

Excreta  disposal  facility  built  in  a  way  that  enables  local  maintenance  over  time 
(Likert  scale  0-4) 

Output 

MOP 

Site  inspection 

Capacity-building 

Community 

Number  of  community  members  trained  in  water  pump  or  latrine/excreta  disposal 

Output 

MOP 

Self-report 

members'  skills  in 
water/sanitation 

maintenance 

maintenance 

Number  of  community  members  trained  to  monitor  water  quality 

Output 

MOP 

Self-report 

improved 

Trained  community  member(s)  maintain  water  pump  or  latrine,  as  required 
(Likert  scale  0-4) 

Outcome 

MOE 

Site  inspection 

Trained  community  member(s)  monitor  water  quality  as  required  (Likert  scale  0-4) 

Outcome 

MOE 

Site  inspection 

NOTE:  The  Likert  scale  assesses  level  of  agreement  with  a  statement.  Here,  0  =  not  at  all,  1  =  small  amount,  2  =  medium  amount,  3  =  large  amount,  4  =  completely. 
a  Nonturbid:  Clear,  not  cloudy,  could  read  a  book  or  newspaper  through  a  tube  of  water. 

b  For  other  types  of  sanitation  projects,  such  as  solid  waste  disposal  or  community  drainage,  other  indicators  will  be  required.  See  the  section  "Choosing  or  Developing 
Additional  Indicators  to  Fill  Project-Specific  Assessment  Gaps,"  later  in  this  chapter. 


Indicators  for  Humanitarian  Assistance  Projects  35 


Table  3.4 

Health  Infrastructure  Project  Indicators 


Project-Specific 

Indicator 

Indicator 

Data 

Strategic  Objective 

Objective 

Indicator 

Type 

Family 

Source 

Humanitarian 

impact 

Health  care 
infrastructure 

Clinic  is  built  or  renovation  complete  (Y/N) 

Output 

MOP 

Self-report 

improved 

New  or  renovated  facility  meets  relevant  local  (and  international  or  national,  as 
appropriate)  building  codes  or  standards  (Y/N) 

Output 

MOP 

Site  inspection 

Facility  built  in  a  way  that  enables  local  maintenance  over  time  (Likert  scale  0-4) 

Output 

MOP 

Site  inspection 

Clinic  is  open  and  being  used  as  intended  (Likert  scale  0-4) 

Outcome 

MOE 

Site  inspection, 

records 

Number  of  people  using  the  new  or  renovated  facility,  divided  by  number  of  people 

Outcome 

MOE 

Site  inspection, 

living  within  5  km  or  1  hour  of  the  facility 

records 

Capacity-building 

Community 
members'  skills 

Number  of  community  members  trained  in  construction  through  the  project 

Output 

MOP 

Self-report 

improved 

NOTE:  The  Likert  scale  assesses  level  of  agreement  with  a  statement.  Here,  0  =  not  at  all,  1  =  small  amount,  2  =  medium  amount,  3  =  large  amount,  4  =  completely. 
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Table  3.5 

Nonhealth  Infrastructure  Project  Indicators 


Project-Specific 

Indicator 

Indicator 

Data 

Strategic  Objective 

Objective 

Indicator 

Type 

Family 

Source 

Humanitarian 

Community 

Facility  is  built  or  renovation  complete  (Y/N) 

Output 

MOP 

Self-report 

impact 

infrastructure 

improved 

New  or  renovated  facility  meets  relevant  local  (and  international  or  national  as 
appropriate)  building  codes  (Y/N) 

Output 

MOP 

Site  inspection 

Facility  built  in  a  way  that  enables  local  maintenance  over  time  (Likert  scale  0-4) 

Output 

MOP 

Site  inspection 

Facility  is  open  and  being  used  as  intended  (Likert  scale  0-4) 

Outcome 

MOE 

Site  visit, 

records 

Number  of  people  using  the  new  or  renovated  facility,  divided  by  number  of  people 

Outcome 

MOE 

Site  visit. 

living  within  5  km  or  1  hour  of  facility3 

records 

Capacity-building 

Community 
members'  skills 

Number  of  community  members  trained  in  construction  through  the  project 

Output 

MOP 

Self-report 

improved 

NOTE:  The  Likert  scale  assesses  level  of  agreement  with  a  statement.  Here,  0  =  not  at  all,  1  =  small  amount,  2  =  medium  amount,  3  =  large  amount,  4  =  completely. 
a  Relevant  to  facilities  serving  the  public,  such  as  schools,  food  distribution  centers,  women's  centers,  or  community  centers. 
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Table  3.6 

Health  Services  Project  Indicators 


Project-Specific 

Indicator 

Indicator 

Data 

Strategic  Objective 

Objective 

Indicator 

Type 

Family 

Source 

Humanitarian 

Access  to  essential 

Health  services  provided  are  the  most  appropriate  and  effective  available  in  the 

Outcome 

MOE 

Site  survey 

impact 

health  services 
increased 

country  and  appropriate  for  the  level  of  facility  (Likert  scale  0-4) 

Percentage  of  community  members  with  access  to  the  health  services  within  S  km 
or  1  hour  travel  time 

Outcome 

MOE 

Site  inspection 

Number  of  persons  receiving  services  in  past  month 

Outcome 

MOE 

Facility  records 

Referral  system  is  in  place  and  operates  effectively  (Likert  scale  0-4) 

Outcome 

MOE 

Site  inspection, 

records 

Capacity-building 

Community 

Local  health  workers  are  integrated  into  the  service  provision  to  the  extent  possible, 

Outcome 

MOE 

Self-report, 

capacity  to  provide 

accounting  for  gender  and  ethnic  balance  (Likert  scale  0-4) 

local 

health  care  services 

consultation 

improved 

Ratio  of  health  care  providers  to  population: 

Number  of  trained  health  care  providers  at  the  project-related  facility  or  facilities 
divided  by  population  size  multiplied  by  100,000  =  ratio  of  providers  per  100,000 
population 

Outcome 

MOE 

Records 

NOTE:  The  Likert  scale  assesses  level  of  agreement  with  a  statement.  Here,  0  =  not  at  all,  1  =  small  amount,  2  =  medium  amount,  3  =  large  amount,  4  =  completely. 
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Table  3.7 

Disaster-Related  Project  Indicators 


Strategic  Objective 

Project-Specific 

Objective 

Indicator 

Indicator 

Type 

Indicator 

Family 

Data 

Source 

Humanitarian 

Disaster 

Major  disaster  risks  determined  (Y/N/NA) 

Input 

MOP 

Self-report 

impact 

preparedness 

improved 

Areas  most  at  risk  determined  (Y/N/NA) 

Input 

MOP 

Self-report 

(if  applicable) 

Disaster  response  plan  developed  (Y/N/NA) 

Output 

MOP 

Self-report 

Number  of  structures  rebuilt,  relocated,  or  retrofitted  to  be  disaster  resistant 

Output 

MOP 

Site  inspection 

Disaster-resistant  structures  located  in  places  to  best  avoid  or  withstand  disaster 
(Y/N/NA) 

Output 

MOP 

Site  inspection 

Number  of  structures  (e.g.,  hospitals,  schools)  with  sufficient  backup  water  and 
power 

Outcome 

MOE 

Site  inspection 

Disaster  response 

There  is  effective  coordination  and  exchange  of  information  among  those  affected 

Process 

MOP 

Site  inspection 

effectively 

coordinated 

by  or  involved  in  the  disaster  response  (Likert  scale  0-4) 

(if  applicable) 

Disaster  recovery 

Contact,  trade,  and  transport  reestablished  between  disaster-affected  rural  areas 

Outcome 

MOE 

Site  inspection, 

achieved 
(if  applicable) 

and  markets  for  products,  labor,  and  services  (Y/N/NA) 

records 

Capacity-building 

Disaster-related 

Project  fills  previously  identified  gap  in  disaster  management  (i.e.,  preparedness, 

Output 

MOP 

Self-report 

capacity  improved 

response,  recovery)  (Likert  scale  0-4) 

Disaster  response  plan  developed  in  collaboration  with  relevant  project  personnel 
or  volunteers  (Y/N/NA) 

Output 

MOP 

Self-report 

Number  of  personnel  or  volunteers  trained  to  implement  disaster  response  plan 

Output 

MOP 

Self-report 

Number  of  personnel  or  volunteers  trained  to  maintain  disaster-resistant 
structures 

Output 

MOP 

Self-report 

Trained  personnel  or  volunteers  maintain  disaster-resistant  structures,  as  required 
(Likert  scale  0-4) 

Outcome 

MOE 

Site  inspection 

NOTE:  The  Likert  scale  assesses  level  of  agreement  with  a  statement.  Here,  0  =  not  at  all,  1  =  small  amount,  2  =  medium  amount,  3  =  large  amount,  4  =  completely. 
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Choosing  or  Developing  Additional  Indicators  to  Fill  Project-Specific 
Assessment  Gaps 

For  many  HA  projects,  the  required  indicators  presented  in  this  chapter  should  be  sufficient  for 
measuring  project  effectiveness.  In  some  cases,  however,  there  may  be  a  unique  set  of  project 
objectives  or  activities  that  are  not  captured  by  the  core  indicators  or  those  related  to  specific 
HA  project  types.  Alternatively,  a  particular  HA  project  may  not  fall  neatly  into  one  of  the 
project  types  (e.g.,  water  and  sanitation,  health  infrastructure)  outlined  here.  In  such  cases,  it 
will  be  important  to  supplement  the  required  MOE  and  MOP  indicators  with  indicators  that 
reflect  the  project’s  effectiveness  relative  to  its  specific,  unique  objectives. 

There  are  key  characteristics  of  good  indicators  that,  if  used  as  guidance,  will  help  ensure 
that  indicators  are  both  usable  and  valuable.  These  characteristics  are  described  in  detail  in  this 
section.  There  are  many  existing  M&E  resources  that  can  be  helpful  in  identifying  appropriate 
indicators  for  many  different  types  of  projects.  The  section  “Resources  for  Identifying  Addi¬ 
tional  Indicators,”  later  in  this  chapter,  provides  a  list  of  relevant  resources  for  this  purpose. 


Characteristics  of  Good  Indicators 

What  are  good  indicators?  Collectively,  good  indicators  are  specific,  measurable,  appropriate, 
reliable,  and  timely.  These  SMART  criteria  are  summarized  in  the  box  below. 


Is  the  indicator  specific? 

Does  the  indicator  clearly  measure  progress  toward  the  result? 

Is  it  clear  what  data  should  be  collected? 

Is  the  indicator  measurable? 

Can  the  necessary  information  be  obtained? 

Are  changes  in  the  indicator  verifiable  by  others  or  through  other  means? 

Is  the  indicator  appropriate? 

Does  the  indicator  sufficiently  capture  project  progress  and  results? 

Are  the  time  and  cost  requirements  for  data  collection  reasonable? 

Will  the  set  of  data  be  useful  to  others  up  the  chain  of  command? 

Is  the  indicator  reliable? 

Is  the  indicator  neutral  and  not  distorted  by  value  judgments? 

Is  the  indicator  able  to  reflect  changing  circumstances  or  situations  (both  positive  and  negative)? 
Is  there  agreement  on  how  the  indicator  should  be  interpreted? 

Is  the  indicator  timely? 

Can  the  indicator  be  collected  in  a  reasonably  timely  fashion? 


Since  food  aid  and  food  security  are  generally  not  priorities  for  OHDACA-funded  HA 
projects,  they  are  not  specifically  covered  in  this  handbook.  Thus,  an  OHDACA  project  that 
focuses  on  this  area  will  require  indicators  not  included  in  the  handbook.  As  an  example, 
consider  an  indicator  for  general  nutritional  support  taken  from  the  Sphere  Project’s  standards 
for  nutrition.  Next,  we  describe  how  one  might  determine  how  well  this  indicator  meets  the 
SMART  criteria  as  a  model  for  how  other  potential  indicators  should  be  assessed. 


•  The  objective  for  general  nutritional  support:  The  nutritional  needs  of  the  population 
are  met. 

•  The  indicator  being  evaluated:  Number  of  people  with  access  to  foods  that  meet  intake 
requirements  of  2,100  kilocalories  per  person  per  day. 
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The  first  question  is  whether  the  indicator  is  specific  enough  to  measure  progress  toward 
the  result.  In  this  example,  the  higher-order  goal  is  to  meet  the  nutritional  needs  of  the  popula¬ 
tion.  Because  this  indicator  reflects  caloric  intake,  it  is  directly  related  to  improving  the  nutri¬ 
tional  needs  of  the  population.  Measuring  change  based  on  this  indicator  will  reflect  changes 
caused  by  the  project.  It  measures  only  one  specific  outcome.  If  protein  intake  were  also  a 
factor,  another  indicator  for  the  percentage  of  calories  from  protein  might  be  implemented. 
While  there  are  other  indicators  that  provide  a  more  sophisticated  picture  of  community  nutri¬ 
tion  that  could  be  added,  caloric  intake  could  stand  alone  as  a  single  specific  indicator  for  com¬ 
munity  nutritional  status  if  necessary.  The  caloric  intake  indicator  is  also  unambiguous.  Most 
development  researchers  would  understand  what  data  should  be  collected  for  this  indicator  and 
what  it  intends  to  measure.  The  use  of  a  target  of  2,100  kilocalories,  suggested  by  the  Sphere 
Project’s  standards,  ensures  that  the  measure  will  be  as  precise  as  possible,  depending  on  how 
data  are  collected. 

The  second  set  of  questions  considers  whether  the  indicator  is  measurable.  That  is, 
can  the  necessary  information  be  obtained?  If  calorie-specific  foodstuffs  are  being  provided, 
then  the  caloric  content  of  those  foods  can  be  determined  directly — whether  or  not  individuals 
are  receiving  2,100  kilocalories  can  be  easily  measured.  If  not,  having  respondents  provide  an 
account  of  foods  consumed  and  calculating  caloric  intake  from  that  would  be  an  appropriate 
proxy,  and  approximate  kilocalorie  intakes  can  be  used  instead.  Another  feature  of  a  measur¬ 
able  indicator  is  that  changes  in  the  variable  can  be  verified  by  others  or  through  other  means. 
In  this  case,  comparing  nutritional  intake  data  with  data  collected  by  other  organizations 
working  on  nutritional  projects  or  conducting  research  in  the  area,  or  comparing  respondents’ 
accounts  to  the  known  caloric  content  of  the  food  being  provided,  would  help  determine 
whether  the  caloric  intake  being  measured  by  the  indicator  is  correct.  The  2,100-kilocalorie 
benchmark  can  be  checked  against  other  sources. 

The  third  set  of  questions  concerns  how  appropriate  the  indicator  is.  For  a  food  distribu¬ 
tion  program,  the  first  priority  is  to  ensure  that  community  members  are  eating  enough  calo¬ 
ries  to  sustain  themselves.  Thus,  measuring  the  number  of  calories  taken  in  and  comparing 
that  to  the  2, 100 -kilocalorie  target  would  be  an  appropriate  measure  of  project  progress  and 
results.  The  indicator  itself  is  an  outcome  measure,  which  can  be  collected  at  baseline,  comple¬ 
tion,  and  one-year  follow-up.  As  mentioned  earlier,  these  data  can  be  collected  in  a  variety  of 
ways,  each  with  different  resource,  time,  and  cost  requirements.  The  method  of  data  collection 
should  be  acceptable  with  respect  to  resources,  time,  and  cost.  For  direct  food  distribution,  the 
additional  costs  would  be  fairly  minor.  For  an  intake  recall  study,  conducting  a  household  or 
community  study  may  be  more  or  less  expensive  than  the  project’s  budget  can  bear.  That  deci¬ 
sion  must  be  made  in  the  context  of  the  project  itself.  Further,  the  timing  of  data  collection 
should  allow  project  managers  to  communicate  results  to  superiors  when  necessary.  It  is  most 
useful  if  the  caloric  intake  data  can  be  collected,  analyzed,  and  summarized  for  superiors  at  a 
time  when  the  data  can  be  used  effectively.  If  resources,  time,  or  costs  of  data  collection  exceed 
the  project  assessment  budget,  or  if  the  data  would  not  be  ready  when  needed  by  superiors, 
then  the  indicator  would  not  be  useful. 

The  fourth  set  of  questions  considers  whether  an  indicator  is  reliable.  The  indicator 
should  be  considered  objective  and  should  be  tested  in  many  contexts.  Because  this  indica¬ 
tor,  2,100-kilocalorie  intake  per  day,  is  suggested  by  the  Sphere  Project,  it  has  been  reviewed 
and  tested  in  a  variety  of  food  distribution  contexts.  There  should  also  be  agreement  on  how 
the  indicator  should  be  interpreted.  The  indicator  must  be  equally  able  to  represent  more  and 
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less  successful  (positive  and  negative)  outcomes,  regardless  of  whether  the  data  are  collected 
from  the  entire  population  or  a  subgroup  of  the  population.  This  indicator  of  caloric  intake  is 
not  biased  for  or  against  any  population  or  subpopulation,  and  the  target  of  2,100  kilocalories 
means  that  positive  (2,100  kilocalories  or  above)  and  negative  (less  than  2,100  kilocalories) 
results  can  be  reported  with  this  indicator.  The  indicator  should  have  no  “built-in”  value  judg¬ 
ment;  this  measure  does  not  favor  any  particular  project.  That  is,  the  indicator  doesn’t  assume 
that  the  project  will  be  successful  or  measure  “how  successful”  the  specific  project  was.  Fur¬ 
ther,  the  indicator  should  be  able  to  record  changing  circumstances  associated  with  the  project. 
Because  the  indicator  is  a  direct  measure  of  caloric  intake,  if  extenuating  circumstances  lead  to 
decreased  caloric  intake  within  a  population,  this  indicator  will  reflect  the  change  and  the  data 
can  be  explained  by  contextual  factors.  With  these  characteristics,  the  indicator  is  as  objective 
and  reliable  as  possible. 

The  fifth  question  is  whether  the  indicator  is  timely:  Can  the  needed  information  be  col¬ 
lected  within  a  reasonable  amount  of  time  (not  posing  undue  burden)  and  as  close  to  project- 
related  time  frames  as  possible?  If  not,  then  it  is  probably  not  a  good  choice,  even  though  it 
might  be  acceptable  in  every  other  way.  In  the  case  of  caloric  intake,  data  can  be  collected 
immediately  or  within  24  hours  of  food  distribution,  if  necessary.  This  is  a  reasonable  amount 
of  time  and  very  close  to  project-related  time  frames,  regardless  of  whether  intake  is  measured 
directly  or  by  intake  recall. 

Resources  for  Identifying  Additional  Indicators 

In  addition  to  the  human  resources  available  through  USAID  and  university  faculty  and  stu¬ 
dents  in  the  host  nation,  there  are  numerous  reference  groups,  projects,  and  documents  that 
can  be  valuable  in  identifying  appropriate  indicators.  The  following  list  of  resources  will  be 
useful  if  it  becomes  necessary  to  identify  new  indicators  for  HA  projects: 

1.  The  Sphere  Project:  The  primary  focus  of  the  Sphere  Project  is  disaster  response,  but 
its  standards  and  indicators  are  applicable  in  nondisaster  contexts  as  well.  The  project 
includes  standards  and  indicators  for  capacity-building;  water  sanitation  and  hygiene; 
food  security,  nutrition,  and  food  aid;  shelter,  settlement  and  nonfood  assistance;  and 
health  services  (http://www.sphereproject.org). 

2.  U.S.  Department  of  State,  Bureau  of  Population,  Refugees,  and  Migration:  In  2007,  this 
office  published  a  standard  “basket  of  indicators,”  which  included  food  security,  nutri¬ 
tion,  and  food  aid;  water,  sanitation,  and  hygiene  promotion;  health;  shelter,  settlement, 
and  nonfood  assistance;  education;  and  self-sufficiency.  Copies  can  be  obtained  by  visit¬ 
ing  http://www.state.gOv/g/prm. 

3.  Core  Group:  This  is  a  collaborative  of  more  than  50  NGO  partners  supported  by  USAID 
whose  goal  is  improving  maternal,  child,  and  community  health.  Its  working  group 
provides  basic  information  on  M&E  and  recommends  standard  indicators  for  projects 
that  focus  on  social  and  behavioral  change,  HIV/AIDS,  malaria,  tuberculosis,  nutri¬ 
tion,  safe  motherhood  and  reproductive  health,  and  integrated  management  of  child¬ 
hood  illness  (http://www.coregroup.org). 

4.  Prevention  Consortium:  The  focus  of  this  consortium  is  community-based  disaster  pre¬ 
vention  and  preparedness.  Its  resources  may  be  helpful  in  designing  disaster-focused 
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projects  and  identifying  related  indicators  (http://www.proventionconsortium.org/ 
?pageid=4). 

5.  United  Nations  International  Strategy  for  Disaster  Reduction  (ISDR):  The  UN’s  ISDR 
Secretariat  focuses  on  disaster  reduction.  Its  mission  is  “building  disaster  resilient  com¬ 
munities  by  promoting  increased  awareness  of  the  importance  of  disaster  reduction  as 
an  integral  component  of  sustainable  development,  with  the  goal  of  reducing  human, 
social,  economic  and  environmental  losses  due  to  natural  hazards  and  related  techno¬ 
logical  and  environmental  disasters”  (http://www.unisdr.org).  Resources  and  publica¬ 
tions  can  be  found  at  http://www.unisdr.org/publications,  and  a  guide  to  indicators 
can  be  found  at  http://www.unisdr.org/eng/about_isdr/isdr-publications/15-indicator- 
of-progress/Indicators_of_Progress_HFA.pdf. 

6.  Food  and  Nutrition  Technical  Assistance  II  Project  (FANTA-2):  This  technical  assis¬ 
tance  group  supported  by  USAID  focuses  on  the  entire  range  of  food-assisted  devel¬ 
opment  projects  and  provides  indicators  for  many  types  of  projects,  including  emer¬ 
gency  and  humanitarian  response,  water  and  sanitation,  urban  food  security,  women’s 
health  and  nutrition,  microcredit,  and  HIV/AIDS.  Its  M&E  guidance  is  applicable 
across  all  types  of  humanitarian  assistance  projects  (http://www.fantaproject.org/focus/ 
monitoring,  shtml) . 

7.  MEASURE  Evaluation  project:  This  USAID-sponsored  technical  assistance  group  works 
to  improve  the  capacity  of  NGOs  and  host-country  programs  to  measure  progress  in 
HA  projects  focused  on  health,  population,  and  poverty  (http://www.cpc.unc.edu/ 
measure).  MEASURE  has  an  extensive  collection  of  publications  describing  how  its 
tools  have  been  applied;  these  publications  can  be  found  at  http://www.cpc.unc.edu/ 
measure/publications.  The  full  set  of  tools,  including  indicators,  can  be  found  at  http:// 
www.cpc.unc.edu/measure/tools. 

8.  International  Fund  for  Agricultural  Development:  This  UN  agency  was  established  after 
the  1974  World  Food  Conference  and  is  dedicated  to  eradicating  rural  poverty,  largely 
through  agricultural  research.  The  agency’s  evaluation  office  has  created  a  very  good 
guide  to  project  M&E  (http://www.ifad.org/evaluation/guide).  The  agency  has  also 
developed  a  comprehensive  results  and  impact  management  framework  (with  software) 
that  focuses  specifically  on  rural  development  projects  and  indicators  for  such  projects: 
http://www.ifad.org/operations/rims. 

9.  Organisation  for  Economic  Co-Operation  and  Development:  The  Development  Co- 
Operation  Directorate  uses  a  “managing  for  development  results”  framework,  with  a 
focus  on  using  indicators  to  determine  progress  (http://www.oecd.org/dac).  The  direc¬ 
torate’s  evaluation-focused  website  is  http://www.oecd.org/dac/evaluationnetwork,  and 
resources  can  be  found  at  http://www.oecd.org/dac/evaluationnetwork/derec. 

10.  World  Bank  Independent  Evaluation  Group:  This  group  works  to  improve  development 
outcomes  by  improving  evaluation.  Its  website  contains  extensive  information  about 
conducting  evaluations  and  many  tools  for  use  in  evaluation  contexts,  particularly  for 
capacity-building  (http://www.worldbank.org/ieg). 

1 1 .  Monitoring,  Evaluation  and  Learning  for  Fragile  States  and  Peacebuilding  Programs:  This 
toolkit,  whose  subtitle  is  “Practical  Tools  for  Improving  Program  Performance  and 
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Results,”  was  developed  in  2005  by  Social  Impact,  Inc.  with  support  from  USAID’s 
Office  of  Transition  Initiatives.  It  provides  a  strong  introduction  to  indicator-driven 
M&E  and  presents  indicators  of  social  change  in  such  areas  as  mitigating  conflict, 
media  and  outreach,  advocacy,  local  governance,  reintegration  of  excombatants,  and 
organizational  capacity-building.  It  also  includes  descriptions  of  methods  suggested  for 
data  collection  and  detailed  examples  of  their  implementation  (http://www.usaid.gov/ 
our_work/cross-cutting_programs/transition_imtiatives).  The  toolkit  can  be  found  at 
http://www.socialimpact.com/resource-center/downloads/fragilestates.pdf. 


Additional  Tips  for  Assessing  Humanitarian  Assistance  Projects 

Proxy  Indicators 

Because  some  direct  measures  are  difficult  or  expensive  to  obtain,  indirect  measures — or  proxy 
indicators — have  proven  quite  useful.  The  following  are  some  examples  of  proxy  indicators 
from  conflict  situations  in  which  such  indicators  have  been  necessary: 

•  Tax  revenues:  In  Afghanistan,  where  the  economy  is  largely  drug- oriented,  tax  revenues 
should  reflect  the  legitimate  economy.  Thus,  increases  in  tax  revenues  suggest  both  the 
reach  of  the  government  and  the  growth  of  the  legitimate  economy. 

•  Number  of  children  in  school:  This  indicator  can  reflect  whether  people  feel  that  it  is  safe 
enough  to  send  their  children  to  school  or  whether  girls  have  more  time  to  attend  school 
because  they  are  no  longer  required  to  spend  a  large  portion  of  the  day  gathering  water. 

•  Markets  open  and  people  shopping:  This  indicator  shows  both  that  people  feel  safe  enough 
to  shop  and  that  there  is  enough  security  for  that  level  of  economic  activity  to  occur. 

Avoiding  Bias 

How  data  are  collected  depends  on  the  data  needed.  One  key  issue  is  to  avoid  systematic  distor¬ 
tion,  or  bias.  Biased  information  can  be  misleading.  Common  sources  of  bias  are  talking  only 
to  men,  talking  only  to  adults,  talking  to  only  one  ethnic  group,  talking  only  to  people  in  safe 
neighborhoods,  and  talking  only  to  local  officials  (who  may  or  may  not  represent  local  citizens). 
In  some  places,  schools  may  provide  counts  of  students  that  include  only  boys.  Whole  studies 
have  been  published  on  conflict  zones  with  a  footnote  stating  that  the  data  were  collected  only 
in  “safe”  neighborhoods,  so  they  probably  did  not  provide  the  best  reflection  of  reality.  There 
are  many  sources  of  bias  that  relate  principally  to  researchers,  informants,  and  data- collection 
tools  and  techniques.  These  types  of  bias  are  described  in  greater  detail  in  Appendix  B.  When 
developing  or  collecting  indicators  as  part  of  project  assessment,  it  is  important  to  recognize 
the  potential  for  and  try  to  avoid  sources  of  bias. 
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General  Tips  for  Avoiding  Bias  in  Project  Assessments 

1 .  Listen  carefully. 

Sometimes,  project  team  members  go  into  the  field  so  focused  on  promoting  their  own  ideas  that  they 
leave  little  time  for  community  members  to  provide  feedback.  Talk  less,  and  listen  more — particularly 
with  respect  to  needs. 

2.  Be  humble. 

Sometimes,  project  team  members  believe  that  their  personal  knowledge  of  the  situation  is  more 
valuable  or  reliable  than  others'  input.  This  makes  community  members  or  other  key  stakeholders  feel 
as  if  their  opinions  don't  count.  Encourage  participation  and  openly  discuss  the  value  of  everyone's 
perspective. 

3.  Let  people  speak  for  themselves. 

When  measurement  includes  community  consultation,  interviews,  or  surveys,  be  sure  to  let  respondents 
speak  for  themselves.  In  some  situations,  project  team  members  are  reluctant  to  hand  over  control  to 
their  respondents.  During  a  survey,  they  ask  the  questions  and  record  respondents'  answers  rather 
than  allowing  each  respondent  to  read  the  survey  and  respond  on  his  or  her  own.  This  sort  of  power 
differential  does  not  encourage  honest  feedback.  When  possible,  let  community  members  take  control 
of  their  own  answers.  This  may  require  translation  into  the  local  language  and  pretesting  for  literate 
populations;  such  an  approach  is  not  feasible  for  nonliterate  populations. 

4.  Keep  an  open  mind. 

Never  assume  that  you  know  the  project  results  before  you  conduct  the  project  assessment.  Project  team 
members,  because  they  have  a  sense  of  ownership  of  the  project,  often  know  what  kinds  of  results  are 
needed  to  show  success — and  sometimes  steer  responses  in  that  direction.  This  is  dishonest  and  doesn't 
allow  for  learning,  improvement,  or  honest  appraisal  of  project  success  or  contribution  to  improved  well¬ 
being  in  a  community.  Avoid  it  at  all  costs. 


PART  II 

Step-by-Step  User's  Guide  for 
Project  Assessment 
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This  user’s  guide  presents  step-by-step  guidance  for  planning  and  implementing  HA  proj¬ 
ect  assessments,  from  beginning  to  end.  It  follows  the  project  cycle  of  planning,  start-up, 
execution,  completion,  and  follow-up,  presented  in  Figure  1.1  in  Chapter  One  of  the  M&E 
Primer  (Part  I  of  this  handbook).  Eleven  steps  are  presented  as  part  of  a  primarily  continuous 
sequence,  categorized  according  to  the  phase  in  the  project  cycle  during  which  they  occur.  A 
summary  of  the  project  assessment  steps  is  presented  in  Table  1  as  a  preview  of  the  step-by- 
step  process  detailed  in  the  remainder  of  this  guide.  The  goal  of  this  guide  is  to  offer  a  brief 
description  of  each  step,  tools  to  help  organize  and  complete  the  steps,  and  tips  for  getting  good 
information  easily  and  effectively.  For  further  details  or  an  explanation  of  any  of  the  steps,  refer 
to  the  M&E  Primer. 


Planning  the  Project  Nomination 

The  project  itself  must  be  planned  before  an  assessment  plan  can  be  designed.  This  section 
describes  the  four  main  steps  involved  in  planning  the  project  nomination,  including  the  use 
of  planning  tools,  where  applicable. 

1.  Consult  with  Relevant  Counterparts  to  Establish  Needs 

The  first  step  for  planning  any  project  is  to  consult  with  relevant  counterparts  to  determine 
needs  and,  thus,  inform  the  development  of  project  objectives.  Such  counterparts  include  other 
members  of  the  country  team  (e.g.,  from  the  Department  of  State  or  USAID),  as  well  as  appro¬ 
priate  representatives  of  the  host  nation,  such  as  the  host-nation  military;  other  government 
agencies  at  the  national,  provincial  or  local  levels;  and  local  civilians  or  civilian  organizations. 
The  best  project  is  one  that  evolves  from  an  unmet  need  from  one  or  more  of  these  groups  and 
involves  them  in  the  planning  of  the  project.  For  example,  what  are  the  unmet  needs  as  deter¬ 
mined  by  USAID,  United  Nations  agencies,  and  the  host  nation?  What  are  the  unmet  needs 
for  the  most  vulnerable  sections  of  the  population?  What  resources  and  expertise  can  DoD 


Table  1 

Summary  of  Steps  for  Planning  and  Carrying  Out  HA  Project  Assessments 


Project  Cycle  Phase 

Project  Assessment  Activities 

Planning  the  project  nomination 

Consult  with  relevant  counterparts  to  establish  needs. 

Design  the  project:  Define  the  project  objectives  and  then  the  desired 
outcomes,  outputs,  activities,  and  inputs. 

Identify  MOPs  and  MOEs. 

Develop  the  timeline  and  budget  for  the  project. 

Planning  the  tactical  details  of 
project  assessment 

Determine  information-gathering  methods. 

Plan  the  logistics  for  project  assessment. 

Start-up:  baseline  measurement 

Collect  selected  baseline  measurements  (MOEs). 

Execution 

Monitor  inputs,  processes,  and  outputs  and  take  corrective  action,  as  needed. 

Completion 

Collect  output  and  outcome  indicators  after  the  project  is  completed. 
Communicate  results. 

Follow-up 

Collect  outcome  indicators  one  year  after  the  project  is  completed,  and 
communicate  results. 
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uniquely  provide  to  address  these  needs?  What  areas  of  need  fall  below  relevant  national  or 
international  standards? 

Consultation  and  coordination  with  USAID  and  Department  of  State  counterparts — 
DoD’s  HA  activities  usually  overlap  with  USAID  and  Department  of  State  development 
strategies — are  important  to  prevent  duplication  of  effort  and  to  ensure  the  most  effective  use 
of  U.S.  government  resources.  Working  alongside  host-nation  counterparts  helps  build  capac¬ 
ity,  an  important  strategic  objective  for  HA  work.  It  also  increases  the  likelihood  that  host- 
nation  representatives  or  community  members  will  feel  that  DoD  or  its  representatives  are 
responsive  to  their  needs,  which  is  likely  to  contribute  to  DoD’s  strategic  objectives  of  access 
and  influence.  Finally,  host-nation  involvement  is  key  to  enhancing  the  legitimacy  of  the  host- 
nation  government  and  other  civilian  organizations  in  the  eyes  of  the  public,  an  important 
component  of  security  cooperation. 

2.  Design  the  Project:  Define  the  Project  Objectives  and  Then  the  Desired  Outcomes, 
Outputs,  Activities,  and  Inputs 

This  step  involves  working  through  the  objective  tree  provided  as  a  blank  template  in  Work¬ 
sheet  1  and  described  in  more  detail  in  Chapter  Two  of  the  M&E  Primer.  An  objective  tree 
helps  articulate  project  objectives  (i.e.,  desired  project  outcomes);  the  inputs,  processes,  and 
outputs  that  are  both  necessary  and  sufficient  for  achieving  those  objectives;  and  higher-order 
program  and  strategic  objectives  (i.e.,  impacts)  to  which  the  project-level  objectives  contribute. 
The  number  of  branches  of  the  objective  tree  should  be  tailored  to  the  scope  of  the  project 
being  planned.  Include  as  many  as  necessary,  but  be  careful  to  define  a  manageable  number. 
Assuming,  for  illustrative  purposes,  that  increasing  access  to  clean  water  is  the  main  objec¬ 
tive  of  a  project  (i.e.,  the  desired  outcome),  the  objective  tree  would  be  completed  by  working 
down  from  that  project  objective  to  determine  how  it  can  be  achieved  through  various  neces¬ 
sary  inputs,  activities,  and  outputs,  and  by  working  up  from  the  objective  to  determine  why 
the  project  objective  is  important  for  achieving  higher-order  strategic  objectives  (i.e.,  impacts). 
Worksheet  1  also  includes  space  for  naming  indicators  (measures  of  performance  [MOPs]  or 
measures  of  effectiveness  [MOEs])  associated  with  each  planned  input,  process,  output,  out¬ 
come,  and  impact.  Identifying  appropriate  MOPs  and  MOEs  is  discussed  in  step  3. 

3.  Identify  Measures  of  Performance  and  Measures  of  Effectiveness 

As  shown  in  Figure  1,  MOPs  refer  to  input,  process,  and  output  indicators,  and  MOEs  refer  to 
outcome  indicators.  Once  the  objective  tree  is  completed,  it  will  be  important  to  ensure  that 
all  inputs,  processes,  outputs,  and  outcomes  are  adequately  addressed  by  at  least  one  indicator 
each.  The  template  in  Worksheet  1  also  includes  space  for  indicators  associated  with  each  ele¬ 
ment  of  the  objective  tree. 

All  HA  projects  will  require  the  collection  of  MOPs  (input,  process,  and  output  indica¬ 
tors)  and  MOEs  (outcome  indicators)  to  assess  achievement  of  both  tactical  (project-level) 
objectives  and  strategic-level  objectives  (e.g.,  access  or  influence,  capacity-building,  posi¬ 
tive  visibility).  Most  of  the  necessary  MOPs  and  MOEs  for  HA  projects  are  provided  in 
Worksheets  2  through  8,  which  can  be  found  at  the  end  of  this  user’s  guide.  The  indicators  are 
divided  into  three  major  categories.  Project  management  indicators  (Worksheet  2)  reflect 
different  aspects  of  performance  and  collaboration  with  other  USG  agencies,  since  national 
and  military  policy  often  require  interagency  cooperation  (e.g.,  in  stability  operations).  The 
standard  project  management  indicators  presented  in  Worksheet  2  should  be  collected  for  all 
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HA  projects.  Core  indicators,  presented  in  Worksheet  3,  should  also  be  collected  for  all  HA 
projects.  Core  indicators  reflect  higher-order  DoD  strategic  objectives  (e.g.,  access,  influence, 
capacity-building)  to  which  all  DoD  HA  projects  should  contribute  to  some  extent.  Indica¬ 
tors  related  to  specific  project  types  (Worksheets  4  through  8)  reflect  typical  project  objec¬ 
tives  for  the  most  common  types  of  OHDACA-funded  HA  projects:  water  and  sanitation 
(Worksheet  4),  health  infrastructure  (Worksheet  5),  nonhealth  infrastructure  (Worksheet  6), 
health  services  (Worksheet  7)  and  disaster-related  initiatives  (Worksheet  8).  These  indicators 
should  be  collected  only  for  HA  projects  of  that  respective  type.  Tables  3.2  through  3.7  in 
Chapter  Three  of  the  M&E  Primer  (Part  I  of  this  handbook)  provide  a  more  detailed  mapping 
of  indicators  in  the  worksheets  to  project-  and  strategic-level  objectives. 

Often,  project  planners  will  want  to  supplement  the  indicators  provided  in  the  worksheets 
with  those  that  reflect  their  own  unique  project  objectives.  When  the  completed  objective  tree 
(Worksheet  1)  indicates  that  additional  indicators  are  warranted,  new  indicators  should  be 
added  to  fully  reflect  what  the  project  is  trying  to  achieve.  The  section  “Characteristics  of  Good 
Indicators”  in  Chapter  Three  of  the  M&E  Primer  describes  how  HA  planners  can  develop  new 
indicators  using  SMART  (specific,  measurable,  appropriate,  reliable,  and  timely)  criteria;  the 
subsequent  section  provides  resources  to  assist  HA  project  planners  in  finding  relevant  existing 
indicators.  Sometimes,  it  is  impossible  to  gather  direct  indicators  for  a  project  outcome.  In  such 
cases,  it  may  be  necessary  to  identify  appropriate  indirect,  or  proxy  indicators,  associated  with 
such  an  outcome.  Chapter  Three  of  the  M&E  Primer  also  provides  a  more  detailed  description 
of  proxy  indicators.  Good  planning  and  a  wise  choice  of  indicators  during  project  development 
will  facilitate  the  entire  project  assessment  process  that  follows. 

4.  Develop  the  Timeline  and  Budget  for  the  Project 

As  always,  a  budget  and  timeline  must  be  developed  for  the  project  as  part  of  the  project  nomi¬ 
nation  process.  When  developing  the  project  budget  and  timeline,  it  is  important  to  factor  in 
the  time  and  resources  required  for  project  assessment.  More  detailed  guidance  on  planning  for 
project  assessment,  including  timeline  and  budgets,  is  presented  in  step  6. 


Planning  for  the  Tactical  Details  of  Project  Assessment 

Once  the  project  itself  has  been  designed,  the  planning  for  project  performance  and  outcome 
monitoring  must  be  completed.  The  following  two  steps  address  the  process  for  planning  the 
tactical  details  of  project  assessment. 

5.  Determine  Information-Gathering  Methods 

The  most  appropriate  methods  for  collecting  assessment  information  should  be  based  on  the 
context,  objectives,  and,  to  a  lesser  extent,  the  size  of  the  study.  Sometimes  (as  suggested  in 
the  worksheets  at  the  end  of  this  user’s  guide  and  in  Chapter  Three  of  the  M&E  Primer), 
there  is  one  main  data  source  or  method  for  collecting  a  particular  MOP  or  MOE.  In  other 
instances,  especially  for  some  MOEs,  there  are  different  approaches.  The  methods  themselves 
can  be  qualitative  or  quantitative.  Here,  the  distinction  is  that  qualitative  data  collection  results 
in  text-based  data — for  example,  a  simple  “yes,”  “no,”  or  “not  applicable”  with  an  accompa¬ 
nying  explanation — while  quantitative  data  collection  includes  counts,  percentages,  or  other 
numbers.  The  data  source  to  be  used  for  collecting  MOP  and  MOE  indicators  is  included  for 
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each  standard  indicator  in  Worksheets  3  through  8.  For  project-specific  indicators,  the  source 
of  data  collection  should  be  filled  in  under  the  worksheet’s  “data  source”  column. 

Next,  each  of  the  different  methods  for  gathering  data  is  briefly  explained.  The  sequenc¬ 
ing  of  the  methods  is  roughly  in  increasing  order  of  complexity  and  cost. 

•  Self-report:  Many  MOPs  relate  to  the  processes  associated  with  project  planning,  execu¬ 
tion,  and  assessment  and  are  a  simple  matter  of  answering  “yes,”  “no,”  or  “not  applicable”; 
some  require  brief  descriptions  (e.g.,  an  explanation  of  problems  encountered,  how  they 
were  resolved  or  why  they  were  not  resolved,  which  U.S.  government  agencies  other  than 
the  State  Department  or  USAID  were  involved  in  the  project  planning). 

•  Records:  Some  indicators  are  based  on  facility  records,  such  as  clinic  records  documenting 
the  number  of  patient  visits. 

•  Chemical  test:  This  method  applies  only  to  certain  types  of  projects,  for  example,  projects 
that  aim  to  create  a  safe  drinking  water  supply,  in  which  chemical  testing  is  used  to  objec¬ 
tively  measure  water  quality. 

•  Site  inspection:  This  method  involves  an  on-site  visit  to  the  project  area  to  directly  observe 
elements  associated  with  one  or  more  indicators  (e.g.,  that  a  well  was  built;  a  facility  is 
operational  and  being  used  as  intended;  the  well  or  facility  meets  relevant  local,  national, 
or  international  standards;  trained  personnel  are  maintaining  the  well  or  facility  as 
planned). 

•  Local  community  consultation:  Consultations  are  usually  carried  out  in  the  project  area 
with  relevant  community  leaders,  beneficiaries  of  the  project’s  outputs,  or  others.  The 
questions  posed  to  individuals  during  interviews  or  discussed  with  a  focus  group  are 
the  relevant  indicators.  For  example,  is  the  host  nation  (military,  nonmilitary,  or  other 
civilian  organization)  willing  to  sustain  the  project  post-completion?  Does  it  have  the 
capacity  to  do  so  (and  what  is  the  evidence  for  this)?  Is  the  project  supported  by  the  local 
community?  How  does  the  community  view  the  quality  of  services?  How  can  services  be 
improved?  Is  DoD  viewed  favorably  by  the  local  community?  Additional  tips  for  good 
community  consultations  are  presented  in  Appendix  A. 

•  Local  community  minisurvey  or  poll:  For  projects  that  are  large  or  in  project  areas  where 
surveying  or  polling  is  already  taking  place  (for  example,  by  the  State  Department  for 
public  diplomacy  purposes),  a  local  survey  or  poll  can  provide  quantitative  information 
that  may  be  more  objective  than  qualitative  information  and  that  may  facilitate  compari¬ 
son  over  time  and  across  projects  and  countries.  A  local  survey  can  be  administered  by  the 
HA  project  overseer  or  a  contractor,  such  as  a  local  university  or  other  technical  contrac¬ 
tor.  Another  option  may  be  to  add  questions  that  are  relevant  to  the  specific  HA  project 
to  a  larger  survey  in  the  area,  such  as  one  that  the  State  Department  may  be  carrying  out, 
if  appropriate.  It  is  always  important  to  target  a  large  enough  sample  to  ensure  statistical 
significance  based  on  estimated  prevalence  and  desired  precision  and  to  use  appropriate 
sampling  techniques. 
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The  specific  steps  for  conducting  minisurveys  or  polls  are  listed  in  Social  Impact,  Inc.’s, 
Monitoring,  Evaluation,  and  Learning  for  Fragile  States  and  Peacebuilding  Programs  toolkit.1 
They  are,  briefly,  as  follows: 

•  Plan  and  get  buy-in  for  the  survey  (for  example,  identify  a  local  university  or  other  con¬ 
tractor  to  administer  the  survey  or  an  ongoing  survey  to  which  study-specific  questions 
can  be  added). 

•  Design  the  questions  (use  the  relevant  indicators — MOPs/MOEs — as  a  guide). 

•  Translate  and  pretest  (depending  on  the  context,  pretesting  the  questions  may  not  be 
necessary  or  practical). 

•  Collect  data. 

—  Identify  the  sampling  frame:  Determine  the  community  from  which  to  draw  the  indi¬ 
viduals  to  be  surveyed,  the  number  of  individuals  to  be  surveyed,  and  how  they  will 
be  selected. 

-  Determine  logistics:  By  whom,  when,  where,  and  how  will  the  survey  interviews  be 
completed? 

-  Consider  biases  (see  Appendix  B):  Try  to  ensure  that  the  individuals  surveyed  are  a  rea¬ 
sonable  representation  of  the  community  of  interest  and  are  not  selected  in  a  way  that 
may  skew  results.  For  example,  it  is  often  important  to  conduct  interviews  with  both 
males  and  females,  all  relevant  age  and  ethnic  groups,  and  all  relevant  socioeconomic 
levels;  the  survey  sample  size  will  need  to  be  larger  if  measuring  statistical  differences 
among  subgroups  is  the  goal. 

•  Analyze  data.  Plan  in  advance  and  then  carry  out  the  analyses,  for  example,  by  mapping 
the  frequency  distribution  of  responses  or  responses  by  gender,  age,  ethnicity,  or  socio¬ 
economic  group.  Relevant  statistical  tests,  such  as  chi-square,  should  be  used  to  assess 
statistical  significance. 

•  Present  findings  (in  a  written  report  or  designated  DoD  system). 

6.  Plan  the  Logistics  for  Project  Assessment 

Logistics  for  project  assessment  involve  several  related  activities: 

Timeline.  To  ensure  that  project  assessment  data  can  be  used  most  effectively  for  both 
the  project  and  any  related  higher-order  evaluations,  it  is  important  that  there  is  a  clear  link 
between  when  indicators  will  be  collected  and  the  project  timeline.  It  is  also  important  to 
ensure  that  superiors  have  an  opportunity  to  review  the  assessment  data  before  accepting 
another  round  of  proposed  projects. 

The  project  timeline  will  be  the  primary  determinant  of  the  timing  of  data  collection  for 
monitoring  purposes.  However,  other  factors  may  drive  the  timeline  for  project  assessment: 
Does  the  COCOM  HA  manager  or  OHDACA  program  have  a  regular  reporting  require¬ 
ment?  Are  there  established  schedules  of  inspection  visits,  of  which  this  might  be  a  small  part? 
When  will  the  data  be  used?  Even  the  best  monitoring  data  are  irrelevant  if  they  come  in  too 
late  to  be  useful. 

Responsible  Party.  The  in-country  HA  project  manager  is  responsible  for  completing 
project  assessment.  However,  others  (for  example,  from  USAID,  an  NGO,  or  a  local  univer- 


1  Social  Impact,  Inc.,  Monitoring,  Evaluation,  and  Learning  for  Fragile  States  and  Peacebuilding  Programs,  Arlington,  Va., 
2005. 
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sity)  might  collaborate  to  collect  monitoring  data.  Will  other  U.S.  country  team  members 
participate?  Any  host-nation  officials?  External  evaluators?  Local  university  faculty?  Before  the 
project  starts,  it  is  important  to  determine  whether  collaborators  will  participate  in  project 
assessment  and,  if  so,  exactly  how.  Worksheets  2  through  8  all  provide  a  column  for  indicating 
who  is  responsible  for  collecting  indicator  information,  and  this  column  should  be  filled  out 
when  planning  for  project  assessment. 

Relevant  Skills.  It  is  in  everyone’s  best  interest  to  ensure  that  the  people  involved  in  the 
assessment  have  the  basic  skills  they  need.  While  counting  the  number  of  people  visiting  a  well 
may  not  be  difficult,  it  must  be  done  consistently,  and  measuring  water  quality  or  flow  from  a 
well  may  require  some  training  to  ensure  that  data  are  collected  correctly.  It  is  important  that 
those  involved  in  project  assessment  know  how  to  collect  the  required  data. 

Data  Entry.  Having  a  clear  idea  of  how  the  data  will  be  entered  before  they  are  collected 
will  make  data  collection  easier  and  more  efficient.  No  matter  who  collects  the  MOPs  and 
MOEs,  the  person  responsible  for  storing  the  information  will  almost  certainly  be  a  DoD  staff 
member  at  the  country  or  COCOM  level.  Worksheets  2  through  8  presented  at  the  end  of 
this  user’s  guide  and  included  in  editable  format  on  the  accompanying  CD  provide  at  least 
an  initial  format  for  recording  and  storing  project  assessment  indicators.  Worksheets  can  be 
printed  out  and  carried  into  the  field  for  the  handwritten  recording  of  information;  later,  the 
information  can  be  captured  more  permanently  by  entering  it  into  the  Overseas  Humanitarian 
Assistance  Shared  Information  System  (OHASIS). 

Resource  Planning  for  Project  Assessment.  The  collection  of  MOPs  and  MOEs  requires 
time,  effort,  and  money.  Without  data  from  MOPs  and  MOEs,  project  managers  and  country 
teams  will  never  be  truly  informed  about  the  results  of  their  projects  and,  thus,  unable  to  learn 
from  successes  and  failures.  It  is  important  to  plan  for  collecting,  analyzing,  and  reporting 
project  MOPs  and  MOEs  in  terms  of  time,  effort,  and  funding.  Planning  in  advance  could 
include  allocating  resources  to  hire  external  personnel  or  strategies  to  collaborate  in  ongoing 
M&E  efforts  of  USG  colleagues  when  their  assessments  can  cover  existing  project  activities. 


Project  Start-Up:  Baseline  Measurement 

7.  Collect  Selected  Baseline  Measurements  (MOEs) 

As  explained  in  Chapter  One  of  the  M&E  Primer,  it  is  important  to  collect  certain  baseline 
measures  before  the  project  is  undertaken.  Specifically,  with  very  few  exceptions,  all  outcome- 
specific  MOEs  should  be  measured  both  before  and  after  project  execution  so  that  changes 
produced  by  the  project  and  the  maintenance  of  those  changes  can  be  documented.  Data  from 
baseline  measurements  should  be  collected  and  entered  into  the  applicable  indicator  work¬ 
sheets.  It  is  useful  to  include  any  observations  of  project-specific  circumstances  (e.g.,  natural 
disasters,  political  schisms)  that  may  affect  project  execution  and  project  effectiveness  before 
the  project  commences.  In  some  cases,  baseline  data  already  exist  and  can  be  taken  from  exist¬ 
ing  databases  maintained  by  the  host  nation,  USAID,  Department  of  State,  or  other  relevant 
and  reliable  source. 
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Project  Execution 

8.  Monitor  Inputs,  Processes,  and  Outputs  and  Take  Corrective  Action,  as  Needed 

For  relatively  small  or  short-term  projects — say,  digging  one  well  or  a  six-month  construction 
project — it  may  or  may  not  be  relevant  to  monitor  project  inputs  and  activities  or  processes 
during  the  project  execution  phase.  Rather,  such  information  can  be  collected  at  the  time  the 
project  is  completed.  For  larger  or  longer-term  projects — multiple  wells  or  a  multiyear  project — 
it  will  almost  certainly  be  appropriate  to  collect  this  information  during  project  execution. 

When  appropriate,  MOPs,  particularly  those  associated  with  completing  projects  on  time 
and  within  budget,  should  be  collected  during  project  execution  (see  Worksheets  2  and  3). 
Ideally,  this  information  will  be  collected,  recorded,  and  used  in  a  timely  fashion  to  make  any 
midcourse  corrections  that  may  be  needed  to  ensure  the  best  possible  project  performance  and 
outcomes.  Collecting  information  during  project  execution  often  provides  a  more  detailed 
explanation  of  successes,  constraints,  and  so  on. 

The  manager  overseeing  the  monitoring  of  inputs  and  activities  or  processes  should  look 
not  only  at  the  progress  of  each  individual  indicator  but  also  at  the  overall  picture.  It  is  not 
uncommon  to  make  good  progress  on  projects  but  to  see  lagging  progress  in  specific  project- 
related  activities.  Problem  solving  midway  through  a  project  could  result  in  suggestions  for 
overcoming  developing  constraints.  Even  if  time  is  too  limited  to  take  corrective  action  during 
a  given  project,  these  findings  can  become  important  lessons  learned  for  other  projects. 


Project  Completion 

9.  Collect  Output  and  Outcome  Indicators  Immediately  After  the  Project  Is  Completed 

Data  should  be  collected  immediately  following  project  completion  and  again  one  year  later. 
Longer-term  follow-up  may  be  required  or  desired  for  larger  or  more  significant  projects,  and 
some  country  teams  choose  to  conduct  formal  or  informal  long-term  follow-up  of  smaller  proj¬ 
ects  after  more  than  one  year.  Output  indicators  (MOPs)  are  collected  upon  project  comple¬ 
tion  and  reflect  the  final  step  of  project  execution — what  is  actually  produced  by  the  project. 
Outcome  indicators  (MOEs)  are  collected  at  baseline,  upon  project  completion,  and  one  year 
later  and  are  chosen  to  reflect  project  effectiveness. 

Immediately  after  project  completion,  it  is  important  to  compare  relevant  MOEs  against 
those  collected  at  project  startup  (baseline  assessment)  to  assess  changes  associated  with  the 
project.  Besides  assessing  change,  this  point  in  the  assessment  cycle  is  a  good  opportunity  to 
assess  important  lessons  learned  that  can  be  incorporated  into  subsequent  projects.  In  some 
cases,  these  lessons  learned  may  be  the  most  important  aspects  of  the  immediate  post-project 
assessment. 

10.  Communicate  Results 

After  the  data  have  been  collected,  they  need  to  be  analyzed,  interpreted,  and  shared  with  col¬ 
laborators  and  supervisors.  In  general,  project  assessment  results  will  be  of  interest  to  the  fol¬ 
lowing  (among  others): 

1.  program  beneficiaries  and  local  leaders 

2.  the  in-country  HA  project  planner,  overseer,  or  manager 
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3.  the  country  team,  especially  USAID  and  relevant  Department  of  State  staff 

4.  the  COCOM  HA  manager 

5.  the  OSD  policy  HA  representative 

6.  the  Defense  Security  Cooperation  Agency,  which  oversees  all  security  cooperation  pro¬ 
grams,  including  OHDACA. 

The  format  for  the  project  reports  should  be  established  in  advance  and  should  commu¬ 
nicate  indicator  information  from  the  various  worksheets.  The  box  below  provides  a  sample 
report  outline.  Items  1  through  7  will  typically  be  completed  during  the  project  nomination 
ph  ase.  Revisions  to  this  information  can  be  made  once  a  project  is  complete  to  more  closely 
reflect  what  actually  occurred.  Items  8,  9,  and  10  reflect  collection  of  MOPs  and  MOEs  at 
project  start-up,  during  project  execution,  immediately  after  project  completion,  and  one  year 
later — all  of  which  should  be  stored  in  OHASIS  for  easy  report  generation. 


Project  Follow-Up 

11.  Collect  Outcome  Indicators  One  Year  After  Project  Is  Completed,  and  Communicate 
Results 

Outcome  measures  should  be  collected  again  one  year  after  the  project  is  completed  to  assess 
the  sustainability  of  the  results.  At  this  point,  the  same  outcome  indicators  (MOEs)  mea¬ 
sured  at  baseline  and  immediately  after  project  completion  should  be  collected  and  recorded 
again.  The  follow-up  outcome  measures  also  should  be  communicated  as  described  in  step  10. 
Sustained  outcomes  are  important  for  the  community  and  contribute  to  the  achievement  of 
broader  strategic  objectives.  Thus,  it  is  important  to  go  back  and  remeasure  outcomes  of  inter¬ 
est  one  year  after  project  completion  and,  potentially  at  future  points,  to  assess  the  sustainabil¬ 
ity  of  outcomes  and  related  lessons  learned. 


Sample  Report  Outline 

1.  Project  title 

2.  Project  location:  COCOM,  country,  project  area  (including  GPS  coordinates  of  the  project) 

3.  Project  start  and  completion  dates 

4.  Total  budget/expenditures 

5.  Introduction  and  background 
— Description  of  need 

— Description  of  context-specific  circumstances  that  may  affect  outcomes 

6.  Project  description 

7.  Project  objectives 

8.  Assessment  of  project  execution: 

— Completion  of  baseline,  immediate  post-project,  and  one-year  post-project  assessments 
— Assessment  methods  used 

— All  MOPs,  reflecting  inputs,  activities/processes,  and  outputs 
— Problems  identified  and  how  they  were  resolved  (or  why  they  were  not) 

9.  Assessment  of  project  outcomes: 

— All  MOEs  reflecting  outcomes  at  baseline,  upon  project  completion,  and  one  year  later 

10.  Conclusions/recommendations/lessons  learned  (to  be  completed  by  user). 
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Handing  Off  Projects 

It  is  important  to  ensure  continuity  of  HA  projects  and  associated  project  assessment  each  time 
staff  turns  over.  This  section  provides  some  suggestions  for  handing  over  a  project  to  a  DoD 
successor  or  other  relevant  agency  that  will  follow  up  on  project  assessment  (collecting  and 
analyzing  MOPs  and  MOEs  for  the  HA  project). 

•  Ongoing  Projects 

-  Provide  information  on  the  status  of  activities  (e.g.,  How  far  along  is  the  project?). 

-  Summarize  required  resources  to  complete  project  assessments  (e.g.,  How  much  time, 
manpower,  or  equipment  will  be  needed  to  complete  the  assessment  process?). 

-  Discuss  key  collaborators  and  important  community  contacts  (e.g.,  Has  the  project 
assessment  successor  been  introduced  to  key  contacts?). 

-  Explain  the  project  assessment  plan  (e.g.,  Have  the  important  indicators  and  methods 
for  collecting  data  been  reviewed  and  explained,  along  with  the  timeline?). 

-  Hand  over  all  indicator  worksheets  and  other  relevant  project  and  project  assessment 
documentation. 

•  Completed  Projects 

-  Review  all  points  discussed  above  for  ongoing  activities  (e.g.,  Does  the  successor  know 
exactly  where  to  find  data  and  reports  for  the  completed  project?). 

-  Explain  what  is  required  for  the  one-year  follow-up  and  discuss  the  follow-up  assess¬ 
ment  plan  (e.g.,  Does  the  successor  know  what  is  involved,  including  when  it  should 
be  conducted,  which  MOE  indicators  need  to  be  collected  and  recorded,  and  how  they 
can  be  compared  to  earlier  measures  of  the  same  MOEs?). 

Using  the  Indicator  Worksheets 

The  following  seven  worksheets  are  designed  to  facilitate  data  collection  and  data  entry.  They 
can  also  be  found  in  electronic  format  on  the  accompanying  CD. 

•  Worksheet  1  presents  the  objective  tree  tool  for  use  in  analyzing  the  causes  and  effects 
related  to  proposed  HA  projects. 

•  Worksheet  2  presents  project  management  indicators  that  are  relevant  to  all  projects. 

•  Worksheet  3  presents  core  indicators  linked  to  DoD  strategic  objectives  that  are  relevant 
to  all  projects. 

•  Worksheet  4  presents  project-specific  indicators  for  water  and  sanitation  projects. 

•  Worksheet  5  presents  indicators  for  health  infrastructure  projects. 

•  Worksheet  6  presents  indicators  for  nonhealth  infrastructure  projects. 

•  Worksheet  7  presents  indicators  for  health  services  projects. 

•  Worksheet  8  presents  indicators  for  disaster-related  projects. 

Worksheets  2  through  8  have  eight  columns  each.  The  first  column  in  each  worksheet 
contains  a  brief  description  of  the  indicator  and  whether  it  is  an  MOP  or  an  MOE.  It  also  pro¬ 
vides  a  definition  of  the  indicator  or  its  calculation.  If  new  indicators  are  developed  or  incor¬ 
porated  into  a  project  assessment,  a  brief  description  of  each  should  be  entered  in  this  column 
under  the  appropriate  heading.  The  second  column  describes  the  method  of  collection  or  the 
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source  of  the  data  for  the  particular  indicator.  As  mentioned  previously,  this  has  already  been 
determined  for  some  indicators.  For  other  indicators,  particularly  those  that  are  specific  to  a 
given  project  type,  it  will  be  necessary  to  enter  the  data  source.  The  third  column  defines  the 
response  options  for  the  indicator.  Again,  these  response  options  are  specified  for  many  indica¬ 
tors  but  will  need  to  be  added  for  others  based  on  the  context  of  a  specific  project. 

In  Worksheet  2,  the  fourth  column  is  intended  for  recording  the  name  of  the  assigned 
data  collector.  In  the  case  of  multiple  data  collectors  at  one  or  more  time  points,  a  note 
should  be  made  to  that  effect  and  additional  documentation  attached  to  the  worksheet.  In 
Worksheets  3  through  8,  the  fourth  column  is  for  the  data  source  actually  used  (the  second 
column  lists  the  possible  data  source).  For  several  indicators,  there  is  more  than  one  potential 
data  source. 

The  fifth,  sixth,  and  seventh  columns  in  Worksheets  2  through  8  are  used  to  record  the 
value  of  the  indicators  at  baseline  (BL),  immediately  upon  completion  (IMM),  and  at  the  one- 
year  follow-up  (1YR).  To  facilitate  data  entry,  when  a  particular  indicator  does  not  need  to  be 
recorded  at  a  certain  time  point,  that  area  of  the  worksheet  is  blacked  out.  The  eighth  column 
is  intended  for  any  notes  related  to  the  particular  indicator.  In  cases  in  which  responses  require 
explanations,  those  should  be  entered  here.  When  extenuating  circumstances  (e.g.,  natural 
disasters  or  political  upheaval)  have  affected  measurement  of  the  indicator,  that  should  be 
noted  here  as  well. 

When  new  indicators  are  needed  for  specific  projects,  follow  the  format  presented  in  these 
worksheets.  That  is,  when  a  scale  or  a  special  type  of  measurement  is  required,  be  sure  to  record 
that  information  in  the  notes  section  or  in  materials  appended  to  the  worksheet.  When  the 
materials  are  taken  from  specific  sources,  be  sure  to  cite  the  source  for  future  reference. 
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Worksheet  1 
Objective  Tree 


Impact 


Outcome 


Output/s 


Processes/ 

activities 


Inputs 


Worksheet  2 

Project  Management  Indicators 


Indicator  (indicator  family:  MOP  or  MOE) 

Data  Source 

Response 

Options 

Response  Value 
for  Specified  Collection  Time 

Data  - 

Collector  BL  IMM  1YR 

Notes/Comments 

Project  planning 

Project  assessment  plan  developed  before  project  execution 
(MOP) 

Self-report 

Y/N 

Post-project  sustainment  plan  developed  (MOP) 

Self-report 

Y/N 

Project  timeline 

Project  completed  by  original  deadline  (MOP) 

Self-report 

Y/N 

(if  no,  explain) 

Project  budget 

Project  completed  within  original  budget  (MOP) 

Self-report 

Y/N 

(if  no,  explain) 

Coordination  across  stakeholder  agencies 

Project  reviewed  and  discussed  with  relevant  State 
Department  officials  and  feedback  incorporated  (MOP) 

Self-report 

Y/N 

Project  reviewed  and  discussed  with  relevant  USAID  officials 
(including  Office  of  Foreign  Disaster  Assistance)  and 
feedback  incorporated  (MOP) 

Self-report 

Y/N/NA 

Project  reviewed  and  discussed  with  other  relevant  USG 
officials  and  feedback  incorporated  (MOP) 

Self-report 

Y/N/NA 

(specify) 

Project  reviewed  and  discussed  with  other  relevant 
agencies  (indicate  which  and  specify  organizations  in  notes/ 
comments  column)  and  feedback  incorporated  (MOP) 

International  organizations 

Self-report 

Y/N/NA 

Other  donor  nations 

Self-report 

Y/N/NA 

Local  and  international  NGOs 

Self-report 

Y/N/NA 

Relevant  private-sector  representatives 

Self-report 

Y/N/NA 

Third-party  allied/coalition  militaries 

Self-report 

Y/N/NA 
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Worksheet  2 — Continued 


Response  Value 

for  Specified  Collection  Time 

Response 

Data  - 

Indicator  (indicator  family:  MOP  or  MOE) 

Data  Source 

Options 

Collector  BL  IMM  1YR 

Notes/Comments 

USAID  provided  financial,  human,  or  in-kind  resources 
(MOP) 

Self-report 

Y/N/NA 

Other  USG  counterpart  (specify)  provided  financial,  human, 
or  in-kind  resources  (MOP) 

Self-report 

Y/N/NA 

Other  USG  counterpart  (specify)  agreed  to  sustain  the 
project  (MOP) 

Self-report 

Y/N/NA 

Host  nation  agreed  to  sustain  the  project  (MOP) 

Self-report 

Y/N/NA 

Project  troubleshooting 


Project  assessed  with  reference  to  stated  objectives  (MOP) 


Baseline  assessment  completed  (MOP) 


Baseline  assessment  conducted  with  at  least  one  community 
member  (e.g.,  leader,  supervisor)  (MOP) 


Immediate  post-project  assessment  completed  (MOP) 


Immediate  post-project  assessment  conducted  with  at  least 
one  community  member  (MOP) 


One-year  post-project  assessment  completed  (MOP) 


Self-report 


Self-report 


Self-report 


Self-report 


Self-report 


Self-report 


Y/N 


Y/N,  date 
completed 


Y/N/NA 


Y/N,  date 
completed 


Y/N/NA 


Y/N,  date 
completed 
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Worksheet  2 — Continued 


Response  Data 

Indicator  (indicator  family:  MOP  or  MOE)  Data  Source  Options  Collector 

Response  Value 
for  Specified  Collection  Time 

BL  IMM  1YR  Notes/Comments 

One-year  post-project  assessment  conducted  with  at  least 
one  community  member  (MOP) 

Self-report 

Y/N/NA 

Additional  post-project  assessment  completed  (date/years 
post-completion)  (MOP) 

Self-report 

Y/N/NA 

NOTE:  BL  =  baseline.  IMM  =  immediate. 
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Worksheet  3 
Core  Indicators 
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Worksheet  3 — Continued 


Indicator  (indicator  family:  MOP  or  MOE) 

Possible  Data  Response 

Source  Options 

Response  Value 
for  Specified  Collection  Time 

Data  Source - 

Used  BL  IMM  1YR 

Notes/Comments 

Project  recognized  as  DoD/USG  effort  and  received  positively  by  the  community 

Project  sends  a  tangible  signal  in  the  host  nation,  regionally, 
and  globally  that  DoD  and  USG  respond  to  humanitarian 
needs  and  have  an  interest  in  the  well-being  of  those  in 
need  (MOP) 

Self-report 

Y/N 

Project  information  recognizing  both  DoD  and  host  nation 
is  disseminated  in  local  community  (MOP) 

Self-report 

Y/N 

DoD  or  other  USG  representative  present  at  the  ribbon¬ 
cutting  or  other  ceremony  (MOP) 

Self-report 

Y/N/NA 

(specify) 

Project  includes,  where  appropriate,  some  tangible,  visible, 
or  substantive  marker  (e.g.,  cornerstone,  plaque,  sign) 
of  both  DoD  and  host-nation  involvement,  with  the  host 
nation  in  the  lead  (MOE) 

Self-report 

Y/N/NA 

(describe) 

Extent  to  which  local  community  feels  project  will  improve 
quality  of  life  (MOE) 

Poll,  local 
consultation 

Likert  scale 
0-4 

Extent  to  which  local  community  feels  the  project  was 
responsive  to  local  needs  (MOE) 

Poll,  local 
consultation 

Likert  scale 
0-4 

Degree  of  favorable  impression  of  the  United  States  (MOE) 

Poll,  local 
consultation 

Likert  scale 
0-4 

Extent  to  which  the  project  changed  community  opinion 
about  the  United  States  and  whether  it  was  in  a  positive  or 
negative  way  (MOE) 

Poll,  local 
consultation 

Likert  scale 
0-4  and  pos/ 
neg 

Cooperated  with  host-nation  civilians  (if  applicable) 


Project  developed  in  response  to  explicit  host-nation  civilian 
need  or  request  (MOP) 

Self-report 

Y/N 

(specify 

organization^]) 

Project  designed  in  coordination  with  host-nation  civilians 
(MOP) 

Self-report 

Y/N 

(specify 

organization^]) 
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Worksheet  3 — Continued 


Response  Value 
for  Specified  Collection  Time 

Possible  Data  Response  Data  Source - 

Indicator  (indicator  family:  MOP  or  MOE)  Source  Options  Used  BL  IMM  1YR  Notes/Comments 

Host-nation  civilians  provided  information  or  financial, 
human,  or  in-kind  resources  for  the  project  (MOP) 

Self-report 

Y/N/NA 

Cooperated  with  host-nation  government  (nonmilitary)  (if  applicable) 

Project  developed  in  response  to  explicit  host-nation 
(nonmilitary)  need  or  request  (MOP) 

Self-report 

Y/N 

(specify 

organization^]) 

Project  designed  in  coordination  with  host-nation 
government  (nonmilitary)  official  (MOP) 

Self-report 

Y/N 

(specify 

organization^]) 

Host  nation  provided  information,  financial,  human,  or 
in-kind  resources  for  project  (MOP) 

Self-report 

Y/N/NA 

Cooperated  with  host-nation  military  for  civilian  benefit  (if  applicable) 

Project  developed  in  response  to  explicit  host-nation 
military  need  or  request  (MOP) 

Self-report 

Y/N 

(specify 

organization^]) 

Project  designed  in  coordination  with  host-nation  military 
officials  (MOP) 

Self-report 

Y/N 

(specify 

organization^]) 

Host-nation  military  provided  information,  financial, 
human,  or  in-kind  resources  (MOP) 

Self-report 

Y/N/NA 

Project  enhanced  host-nation  government  legitimacy  in  the  eyes  of  the  public 

At  least  one  host-nation  government  official  (military  or 
nonmilitary)  present  at  the  groundbreaking  or  ribbon¬ 
cutting  ceremony  (MOP) 

Self-report 

Y/N 

Project  outputs  are  operated,  maintained,  or  managed  by 
the  host-nation  government  (military  or  nonmilitary)  (MOE) 

Self-report 

Y/N/NA 

NOTE:  BL  =  baseline.  IMM  =  immediate. 

Likert  scale:  0  =  not  at  all,  1  =  small  amount,  2  =  medium  amount,  3  =  large  amount,  4  =  completely. 
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Worksheet  4 

Indicators  for  Water  and  Sanitation  Projects 


Possible  Data 

Indicator  (indicator  family:  MOP  or  MOE)  Source 

Response 

Options 

Response  Value 
for  Specified  Collection  Time 

Data  Source - 

Used  BL  IMM  1YR 

Notes/Comments 

Access  to  improved  source  of  drinking  water  enhanced  (if  applicable) 

Number  of  potable  water  sources  created  (with  locations 
indicated  under  "Notes/Comments")  (MOP) 


Well  built  to  relevant  local  (and  international  or  national,  as 
appropriate)  standards  (MOP) 


Well  built  in  a  way  that  enables  local  maintenance  over  time 
(MOP) 


Number  of  people  obtaining  water  from  source  daily, 
divided  by  number  of  people  living  within  500  m  or  a 
10-minute  walk  of  safe  drinking  water  source  (MOE) 


Self-report, 

geocoded 

data 


Self-report 


Site 

inspection 


Site 

inspection, 

household 

survey 


Y/N 


Likert  scale 
0-4 


(numerator), 

# 

(denominator), 

percentage 


Water  quality  enhanced  (if  applicable) 


Drinking  water  from  source(s)  created  by  project  is 
nonturbid  by  inspection3  (MOE) 

Site 

inspection 

Y/N 

Number  of  fecal  coliforms  per  100  ml  at  distribution  point 
(MOE) 

Chemical  test 

# 

Number  of  fecal  coliforms  per  100  ml  at  point  of  use  (MOE) 

Chemical  test 

# 

Access  to  excreta  disposal  facilities  increased  (if  applicable)13 


Number  of  latrines  dug  or  other  excreta  disposal  facilities 
created  (MOP) 

Self-report 

# 

Number  of  people  using  excreta  disposal  daily,  divided  by 
number  of  people  living  within  50  m  of  adequate  excreta 
disposal  (MOE) 

Self-report 

# 

(numerator), 

# 

(denominator), 

percentage 
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Worksheet  4 — Continued 


Response  Value 

for  Specified  Collection  Time 

Possible  Data  Response 

Data  Source - 

Indicator  (indicator  family:  MOP  or  MOE) 

Source  Options 

Used  BL  IMM  1YR 

Notes/Comments 

Excreta  disposal  facility  created  meets  relevant  local  (and 
international  or  national,  as  appropriate)  standards  of 
quality  (MOE) 

Self-report 

Y/N 

Excreta  disposal  facility  built  in  a  way  that  enables  local 
maintenance  over  time  (MOP) 

Site 

inspection 

Likert  scale 
0-4 

Community  members'  skills  in  water/sanitation  maintenance  improved 


Number  of  community  members  trained  in  water  pump  or 
latrine/excreta  disposal  maintenance  (MOP) 

Self-report 

# 

Number  of  community  members  trained  to  monitor  water 
quality  (MOP) 

Self-report 

# 

Trained  community  member(s)  maintain  water  pump  or 
latrine  as  required  (MOE) 

Site 

inspection 

Likert  scale 
0-4 

Trained  community  member(s)  monitor  water  quality  as 
required  (MOE) 

Site 

inspection 

Likert  scale 
0-4 

NOTE:  BL  =  baseline.  IMM  =  immediate. 

Likert  scale:  0  =  not  at  all,  1  =  small  amount,  2  =  medium  amount,  3  =  large  amount,  4  =  completely. 
a  Nonturbid:  Clear,  not  cloudy,  could  read  a  book  or  newspaper  through  a  tube  of  water. 

b  For  other  types  of  sanitation  projects,  such  as  solid  waste  disposal  or  community  drainage,  other  indicators  will  be  required,  see  the  section  "Choosing  or  Developing 
Additional  Indicators  to  Fill  Project-Specific  Assessment  Gaps"  in  Chapter  Three  of  the  M&E  Primer  (Part  I  of  this  handbook). 
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Worksheet  5 

Indicators  for  Health  Infrastructure  Projects 


Response  Value 
for  Specified  Collection  Time 

Possible  Data  Response  Data  Source  - 

Indicator  (indicator  family:  MOP  or  MOE)  Source  Options  Used  BL  IMM  1YR  Notes/Comments 

Health  care  infrastructure  improved 

Clinic  is  built  or  renovation  complete  (MOP) 

Self-report 

Y/N 

New  or  renovated  facility  meets  relevant  local  (and 
international  or  national,  as  appropriate)  building  codes  or 
standards  (MOP) 

Site 

inspection 

Y/N 

Facility  built  in  a  way  that  enables  local  maintenance  over 
time  (MOP) 

Site 

inspection 

Likert  scale 
0-4 

Clinic  is  open  and  being  used  as  intended  (MOE) 

Site 

inspection, 

records 

Likert  scale 
0-4 

Number  of  people  using  the  new  or  renovated  facility, 
divided  by  number  of  people  living  within  5  km  or  1  hour 
of  the  facility  (MOE) 

Site 

inspection, 

records 

# 

(numerator), 

# 

(denominator), 

percentage 

Community  members'  skills  improved 

Number  of  community  members  trained  in  construction 
through  the  project  (MOP) 

Self-report 

# 

NOTE:  BL  =  baseline.  IMM  =  immediate. 

Likert  scale:  0  =  not  at  all,  1  =  small  amount,  2  =  medium  amount,  3  =  large  amount,  4  =  completely. 
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Worksheet  6 

Indicators  for  Nonhealth  Infrastructure  Projects 


Response  Value 
for  Specified  Collection  Time 

Possible  Data  Response  Data  Source  - 

Indicator  (indicator  family:  MOP  or  MOE)  Source  Options  Used  BL  IMM  1YR  Notes/Comments 

Community  infrastructure  improved 

Facility  is  built  or  renovation  complete  (MOP) 

Self-report 

Y/N 

New  or  renovated  facility  meets  relevant  local  (and 
international  or  national  as  appropriate)  building  codes 
(MOP) 

Site 

inspection 

Y/N 

Facility  built  in  a  way  that  enables  local  maintenance  over 
time  (MOP) 

Site 

inspection 

Likert  scale 
0-4 

Facility  is  open  and  being  used  as  intended  (MOE) 

Site  visit, 
records 

Likert  scale 
0-4 

Number  of  people  using  the  new  or  renovated  facility, 
divided  by  the  number  of  people  living  within  5  km  or 

1  hour  of  facility3  (MOE) 

Site  visit, 
records 

# 

(numerator), 

# 

(denominator), 

percentage 

Community  members'  skills  improved 

Number  of  community  members  trained  in  construction 
through  the  project  (MOP) 

Self-report 

# 

NOTE:  BL  =  baseline.  IMM  =  immediate. 

Likert  scale:  0  =  not  at  all,  1  =  small  amount,  2  =  medium  amount,  3  =  large  amount,  4  =  completely. 


a  Relevant  to  facilities  serving  the  public,  such  as  schools,  food  distribution  centers,  women's  centers,  or  community  centers. 
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Worksheet  7 

Indicators  for  Health  Services  Projects 


Indicator  (indicator  family:  MOP  or  MOE) 

Possible  Data 
Source 

Response 

Options 

Data  Source 
Used 

Response  Value 
for  Specified  Collection  Time 

BL  IMM  1YR 

Notes/Comments 

Access  to  essential  health  services  increased 

Health  services  provided  are  the  most  appropriate  and 
effective  available  in  the  country,  appropriate  for  the  level 
of  facility  (MOE) 

Site  survey 

Likert  scale 
0-4 

Percentage  of  community  members  with  access  to  the 
health  services  within  5  km  or  1  hour  travel  time  (MOE) 

Site 

inspection 

Percentage 

Number  of  persons  receiving  services  in  past  month  (MOE) 

Facility 

records 

# 

Referral  system  is  in  place  and  operates  effectively  (MOE) 

Site 

inspection, 

records 

Likert  scale 
0-4 

Community  capacity  to  provide  health  care  services  improved 


Local  health  workers  are  integrated  into  the  service 
provision  to  the  extent  possible,  accounting  for  gender  and 
ethnic  balance  (MOE) 

Self-report, 

local 

consultation 

Likert  scale 
0-4 

Ratio  of  health  care  providers  to  population: 

Number  of  trained  health  care  providers  at  the  project- 
related  facility  or  facilities  divided  by  population  size 
multiplied  by  100,000  =  ratio  of  providers  per  100,000 
population  (MOE) 

Records 

#/1 00,000 

NOTE:  BL  =  baseline.  IMM  =  immediate. 

Likert  scale:  0  =  not  at  all,  1  =  small  amount,  2  =  medium  amount,  3  =  large  amount,  4  =  completely. 
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Worksheet  8 

Indicators  for  Disaster-Related  Projects 


Response  Value 
for  Specified  Collection  Time 

Possible  Data  Response  Data  Source  - 

Indicator  (indicator  family:  MOP  or  MOE)  Source  Options  Used  BL  IMM  1YR  Notes/Comments 

Disaster  preparedness  improved 

Major  disaster  risks  determined  (MOP) 

Self-report 

Y/N/NA 

Areas  most  at  risk  determined  (MOP) 

Self-report 

Y/N/NA 

Disaster  response  plan  developed  (MOP) 

Self-report 

Y/N/NA 

Number  of  structures  rebuilt,  relocated,  or  retrofitted  to  be 
disaster  resistant  (MOP) 

Site 

inspection 

# 

Disaster-resistant  structures  located  in  places  to  best  avoid 
or  withstand  disaster  (MOP) 

Site 

inspection 

Y/N/NA 

Number  of  structures  (e.g.,  hospitals,  schools)  with  sufficient 
backup  water  and  power  (MOE) 

Site 

inspection 

# 

Disaster  response  effectively  coordinated 

There  is  effective  coordination  and  exchange  of  information 
among  those  affected  by  or  involved  in  the  disaster 
response  (MOP) 

Site 

inspection 

Likert  scale 
0-4 

Disaster  recovery  achieved 

Contact,  trade,  and  transport  reestablished  between 
disaster-affected  rural  areas  and  markets  for  products, 
labor,  and  services  (MOE) 

Site 

inspection, 

records 

Likert  scale 
0-4 

Disaster-related  capacity  improved 


Project  fills  previously  identified  gap  in  disaster 
management  (i.e.,  preparedness,  response,  recovery)  (MOP) 

Self-report 

Likert  scale 
0-4 

Disaster  response  plan  developed  in  collaboration  with 
relevant  local  project  personnel  or  community  volunteers 
(MOP) 

Self-report 

Y/N/NA 

Number  of  personnel  or  volunteers  trained  to  implement 
community  disaster  response  plan  (MOP) 

Self-report 

# 
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Worksheet  8 — Continued 


Indicator  (indicator  family:  MOP  or  MOE) 

Possible  Data 
Source 

Response 

Options 

Data  Source 
Used 

Response  Value 
for  Specified  Collection  Time 

BL  IMM  1YR 

Notes/Comments 

Number  of  personnel  or  volunteers  trained  to  maintain 
disaster-resistant  structures  in  the  community  (MOP) 

Self-report 

# 

Trained  personnel  or  volunteers  maintain  disaster-resistant 
structures,  as  required  (MOE) 

Site 

inspection 

Likert  scale 
0-4 

NOTE:  BL  =  baseline.  IMM  =  immediate. 

Likert  scale:  0  =  not  at  all,  1  =  small  amount,  2  =  medium  amount,  3  =  large  amount,  4  =  completely. 
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APPENDIX  A 


Guide  to  Community  Consultation 


The  following  tips  from  USAID  outline  good  practices  for  consulting  with  the  community  for 
purposes  of  project  planning,  execution,  and  assessment.1 

1.  Consult  with  relevant  U.S.  government  staff  in  country. 

Meet  with  relevant  USAID  program  officers)  and  Department  of  State  refugee  coordi¬ 
nator,  if  relevant,  to 

-  discuss  the  OHDACA  program  and  the  specific  proposed  project 

-  ensure  that  the  project  meets  unmet  needs  with  no  duplication  of  effort 

-  coordinate  field  visits 

-  understand  sensitivities  among  the  population  and  HA  community  regarding  DoD’s 
HA  activities. 

After  consultations  with  USG  agencies,  consult  with  relevant  persons  from  the  host 
nation,  UN  agencies,  and  NGOs  to  discuss  these  same  issues. 

2.  Select  the  community  consultation  team. 

The  team  should  include,  at  minimum,  a  facilitator  and  a  note-taker.  The  facilitator 
should  be  a  native  speaker  with  good  skills  in  leading  group  discussions.  The  facilitator 
should  also  have  reasonable  knowledge  of  the  local  area.  Sometimes,  it  is  important  for 
the  facilitator  to  be  of  the  same  gender  as  participants.  This  may  be  possible  for  indi¬ 
vidual  interviews  but  less  practical  for  group  discussions  unless  such  discussions  are 
organized  by  gender. 

3.  Identify  the  participants  for  each  consultation  discussion. 

Based  on  knowledge  of  the  local  situation  and  on  previous  consultations,  identify 
the  individuals,  groups,  and  institutions  that  should  be  represented.  One  of  the  best 
approaches  is  to  consult  several  key  informants.  For  example,  talk  to  NGOs  working 
in  the  area,  to  USAID  and  other  USG  agencies,  or  to  someone  at  a  relevant  local  uni¬ 
versity  who  is  familiar  with  the  situation.  Consulting  several  informants  will  help  avoid 
bias  and  ensure  that  the  correct  individuals  have  been  identified. 

-  Some  issues  to  consider  when  choosing  representatives  for  inclusion  in  the  discussion 
are  whether  it  is  important  to  involve  community  leaders  (both  secular  and  religious), 
government  and  nongovernmental  organizations,  individuals  of  various  ages  (as  rele¬ 
vant  for  the  particular  project),  both  women  and  men,  those  considered  to  be  more  and 


1  Adapted  from  U.S.  Agency  for  International  Development,  1996. 
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less  powerful  in  the  community,  or  all  relevant  ethnic  groups.  In  some  cases,  several 
group  consultations  will  be  necessary  to  gain  the  required  feedback. 

-  Each  group  consultation  or  discussion  should  include  no  more  than  seven  to  11  people, 
to  allow  the  smooth  flow  of  conversation. 

-  Participants  in  each  group  should  be  fairly  homogeneous  to  facilitate  discussion. 
Women  tend  to  defer  to  men.  In  ethnic  conflicts,  certain  groups  may  not  be  comfort¬ 
able  together. 

4.  Determine  timing  and  location. 

Consultations  should  last  no  more  than  one  to  two  hours  and  should  be  conducted 
in  a  convenient  location  with  some  degree  of  privacy.  Consultations  in  village  squares 
may  draw  attention  and  uninvited  guests.  Consultations  in  difficult-to-reach  location  or 
“elite”-feeling  locations,  such  as  hotels,  may  keep  some  individuals  from  participating. 

5.  Specify  the  goals  of  the  consultation. 

What  specifically  should  be  learned  in  this  consultation?  Some  possible  goals  might  be 

-  finding  out  what  types  of  activities  the  community  members  need  or  want 

-  having  community  members  provide  their  opinion  on  which  of  two  or  more  possible 
projects  is  more  relevant  (i.e.,  having  them  help  choose  between  an  elementary  school 
and  a  vocational  school) 

-  getting  their  help  in  determining  the  best  location  for  the  project 

-  asking  their  opinion  of  the  best  time  to  start  the  project 

-  gauging  the  extent  to  which  the  community  is  willing  to  participate  and  what  the 
population  might  be  willing  to  do. 

If  necessary,  clarify  that  the  consultation  concerns  a  very  focused  and  limited  project  to 
be  funded  by  USG.  Unrealistically  high  expectations  on  the  part  of  the  community  will 
ultimately  result  in  disappointment,  regardless  of  the  project  chosen.  Explain  whether 
it  will  be  a  one-time  project  or  whether  it  is  the  first  of  several.  Staff  turnover  is  often 
high,  so  be  very  careful  not  to  promise  something  that  a  different  team  may  not  be  able 
to  deliver. 

6.  Manage  the  flow  of  the  consultation. 

People  may  not  know  what  to  expect  from  a  discussion  of  this  type.  First,  clarify  the 
reason  for  the  consultation  and  tell  participants  that  it  is  an  informal  discussion  in 
which  everyone  is  expected  and  encouraged  to  participate,  especially  if  they  have  diver¬ 
gent  views.  During  the  discussion,  be  careful  about  using  “why”  questions,  which  can 
put  people  on  the  defensive  and  cause  them  to  give  politically  correct  or  polite  answers. 
Instead,  use  probing  techniques  to  get  a  fuller  picture: 

-  Repeat  the  question  to  give  the  participant  more  time  to  think. 

-  Pause  after  an  answer;  be  patient  and  wait  for  a  response.  A  thoughtful  nod  or  expect¬ 
ant  look  can  convey  that  you  want  a  fuller  answer. 

-  Repeat  the  reply;  sometimes,  this  stimulates  conversation. 

-  Ask  “when,”  “what,”  “where,”  and  “which”  questions  to  get  more  detailed  information 
from  participants. 

•  When  is  the  best  time  to  start  construction  on  the  well? 
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•  What  might  happen  if  construction  were  delayed  for  some  reason? 

•  Where  do  you  think  the  well  should  be  located? 

•  Which  neighborhoods  would  benefit  the  most  from  the  well  being  dug  there? 

-  Use  neutral  questions  (e.g.,  “Anything  else?”)  to  get  more  information. 

Sometimes  it  is  important  to  take  control  of  the  discussion.  In  most  groups,  a  few 
individuals  tend  to  dominate  the  discussion.  However,  there  are  ways  to  balance  out 
participation: 

-  Address  questions  to  those  who  are  reluctant  to  talk. 

-  Give  nonverbal  cues.  For  example,  look  in  another  direction  when  someone  talks  for  an 
extended  period.  This  is  a  sign  that  you  would  like  them  to  wrap  up  their  comments. 

-  If  necessary,  intervene,  politely  summarize  the  point,  then  refocus  the  discussion. 

-  Take  advantage  of  natural  pauses.  Say,  “Thank  you  for  that  interesting  idea,”  and  move 
on. 

Minimize  the  effects  of  group  pressure.  When  an  idea  is  adopted  by  the  group  without 
any  general  discussion  or  disagreement,  it  is  likely  the  result  of  group  pressure.  If  this 
happens,  probe  for  more  details  or  alternate  views. 

7.  Analyze  results. 

After  each  interview,  summarize  the  information  gathered  from  the  discussion  and  the 
team’s  impressions  of  the  discussion  in  relation  to  the  project  or  activity  being  consid¬ 
ered.  After  all  consultations  have  been  completed,  review  all  the  information,  analyze 
patterns  and  trends,  and  use  that  information  to  help  identify  options  for  projects  and 
activities  in  the  context  of  relevant  guidance  and  the  appropriate  campaign  plans. 
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Avoiding  Bias 


This  appendix  provides  more  detail  on  the  different  kinds  of  bias  that  may  arise  during  project 
assessment.  Understanding  these  sources  of  bias  will  help  planners  avoid  them. 


Researcher  Bias 

Every  individual  who  collects  qualitative  and  quantitative  data  will  view  things  slightly  differ¬ 
ently.  A  physician  measuring  the  effectiveness  of  a  water  and  sanitation  project  would  be  more 
likely  to  consider  waterborne  illness  and  might  focus  heavily  on  that  set  of  indicators.  An  engi¬ 
neer  might  focus  on  the  amount  of  water  made  available  via  the  pump.  It  is  useful  for  more 
than  one  person  to  conduct  the  important  parts  of  the  assessment  to  offset  this  type  of  bias  by 
presenting  differing  viewpoints  of  the  same  project. 

Researchers  may  also  feel  more  comfortable  speaking  to  a  particular  segment  of  the  popu¬ 
lation,  based  on  gender,  ethnicity,  language  skills,  and  so  on.  It  is  important  to  work  against 
that  “comfort-based”  bias  as  much  as  possible.  When  working  with  local  researchers  as  evalua¬ 
tion  partners,  consider  the  extent  to  which  consultants,  university  professors,  and  even  gradu¬ 
ate  students  are  elites  and  may  not  communicate  well  with  villagers.  Also  watch  for  systematic 
distortion  of  results,  or  bias,  related  to  gender,  since  consultants  or  university  professors  are 
likely  to  be  men  and  may  prefer  to  speak  with  men. 


Informant  Bias 

The  people  that  provide  input  for  polls,  surveys,  focus  groups,  interviews,  and  most  other  data- 
collection  approaches  used  for  project  assessment  also  bring  their  own  biases  into  the  processes. 
Women,  who  are  the  primary  visitors  to  a  well,  will  likely  have  a  very  different  perspective  on 
the  well’s  utility  to  the  village  than  would  men.  This  is  true  across  a  number  of  segments.  It  is 
important  to  gather  input  from  each  of  the  important  segments  of  the  population  to  make  sure 
that  the  data  are  as  representative  of  the  entire  community  as  possible. 

1.  Gender  bias:  The  perspectives  of  women  differ  from  those  of  men  in  some  circumstances. 

2.  Age  bias:  The  perspectives  of  children  or  the  elderly  may  differ  from  those  of  adults  in 
some  circumstances. 

3.  Ethnic  bias:  Certain  ethnic  groups,  particularly  minorities,  may  have  different  perspec¬ 
tives. 
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4.  Spatial  bias:  The  opinions  of  people  who  live  closer  to  the  project  differ  from  those  who 
are  more  remote.  Opinions  in  cities  may  be  different  from  opinions  in  rural  areas. 

5.  Wealth  bias:  Opinions  of  the  wealthy  or  powerful  may  differ  from  those  of  the  middle 
class,  which,  in  turn,  may  differ  from  those  of  the  poor. 

6.  Education  bias:  Perspectives  may  differ  between  people  with  more  or  less  education. 
Further,  because  those  with  more  education  may  be  better  able  to  communicate,  their 
views  may  be  easier  to  understand. 

7.  Expectation  bias:  A  respondent’s  expectations  about  what  the  project  team  could  have 
done  compared  to  what  it  did  may  affect  perspectives.  For  example,  a  villager  may 
believe  that  the  United  States’  vast  wealth  means  that  it  should  have  built  an  entirely 
new  hospital  rather  than  refurbishing  an  existing  clinic  in  a  village. 

8.  Disciplinary  bias:  The  perspective  of  a  village  doctor  may  differ  from  that  of  a  village 
farmer,  for  example. 

9.  Insider-outsider  bias:  Perspectives  may  differ  between  those  who  are  well  integrated  into 
the  community  and  those  who  are  peripheral  to  the  community.  This  may  be  related  to 
ethnicity,  religion,  language,  citizenship,  or  other  characteristics. 


Bias  Related  to  Tools  and  Techniques  Used  to  Gather  Data 

Survey  data  may  provide  a  very  different  picture  compared  with  interview  or  focus  group  data. 
Surveys,  because  they  focus  on  asking  many  easy-to-answer  questions,  may  provide  data  that 
are  more  “shallow”  than  would  be  the  case  in  an  in-depth  interview  with  an  individual.  Fur¬ 
ther,  focus  groups,  because  they  involve  groups  of  people,  can  often  encourage  people  to  report 
what  they  believe  community  members  would  like  to  hear  rather  than  their  personal  opinions. 
Each  approach  is  useful  for  different  goals;  combined,  they  provide  a  better  overall  picture  of 
project  results. 

When  the  data  are  collected  may  also  have  an  effect.  The  time  of  the  year  and  the  time 
of  day  become  important,  depending  on  the  work  schedules  and  livelihoods  of  community 
members.  Farmers  may  not  have  much  time  to  spare  while  planting  or  harvesting;  in  a  village 
populated  largely  by  farmers,  data  collected  at  that  time  may  be  less  reliable. 

Finally,  bias  may  be  introduced  when  people  change  behaviors  of  interest  merely  because 
they  are  being  studied.  This  is  called  observation  bias ,  or  the  “Hawthorne  effect.”1 


1  The  term  Hawthorne  effect  was  coined  in  the  1950s  by  Henry  Landsberger,  who  was  analyzing  experiments  on  worker 
productivity  at  the  Hawthorne  Works  (a  Western  Electric  factory).  His  analysis  showed  that  improvements  in  produc¬ 
tivity  resulting  from  changes  in  working  conditions  disappeared  when  the  study  was  completed,  suggesting  that  the 
productivity  gains  were  attributable  to  the  interest  in  the  workers  being  shown  by  the  study  team  rather  than  the  actual 
changes  to  the  working  conditions. 


APPENDIX  C 

Monitoring  and  Evaluation  Terms 


While  the  list  is  not  exhaustive,  this  glossary  provides  a  more  complete  set  of  important  M&E 
terms  based  on  glossaries  or  dictionaries  developed  by  the  Department  of  State,  the  United 
Nations,  USAID,  and  others.  The  glossary  presents  relevant  definitions  and  the  sources  for 
each  definition. 


Term 

Definition 

Activity 

1.  An  action  or  process  undertaken  over  a  specific  period  to  convert  resources  to  products 
or  services  to  achieve  results.3 

2.  Actions  undertaken  or  work  performed  through  which  inputs,  such  as  funds,  technical 
assistance,  or  other  types  of  resources  are  mobilized  to  produce  specific  outputs.13 

Related  Terms:  project,  development  intervention 

Assessment 

A  process  (which  may  or  may  not  be  systematic)  of  gathering  information,  analyzing  it, 
and  making  a  judgment  based  on  the  information.1- 

Baseline 

Information  collected  before  or  at  the  start  of  a  project  or  program  that  provides  a  basis 
for  planning  or  assessing  subsequent  progress  and  impact.3 

Related  term:  benchmark 

Benchmark 

1.  A  standard  against  which  results  are  measured.3 

2.  Reference  point  or  standard  against  which  performance  or  achievements  can  be 
compared.  A  benchmark  might  refer  to  what  has  been  achieved  in  the  past,  e.g.,  by 
comparable  organizations,  or  what  could  reasonably  have  been  achieved  under  the 
circumstances. c 

Related  term:  baseline 

Data 

Information  collected  by  a  researcher.  Data  gathered  during  an  evaluation  are 
manipulated  and  analyzed  to  yield  findings  that  serve  as  the  basis  for  conclusions  and 
recommendations.3 

Effect 

Intended  or  unintended  change  that  results  directly  or  indirectly  from  an  intervention.3 

Related  terms:  result,  outcome,  impact 

Effectiveness 

1.  The  extent  to  which  an  intervention  has  attained  its  major  relevant  objectives.3 

2.  The  extent  to  which  the  intervention's  objectives  were  achieved  or  are  expected  to 
be  achieved,  taking  into  account  their  relative  importance.  Effectiveness  is  also  used 
as  an  aggregate  measure  of  (or  judgment  about)  the  merit  or  worth  of  an  activity  (i.e., 
the  extent  to  which  an  intervention  has  attained  or  is  expected  to  attain  its  major 
relevant  objectives  efficiently,  in  a  sustainable  fashion,  and  with  a  positive  institutional 
development  impact). b 

End  state 

The  set  of  required  conditions  that  define  achievement  of  the  commander's  objectives.01 
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Term 


Evaluation 


Feedback 


Goal 


Impact 


Impact  evaluation 


Indicator 


Definition 


1.  A  systematic  and  objective  assessment  of  an  ongoing  or  completed  project,  program, 
or  policy.  Evaluations  are  undertaken  to  (a)  improve  the  performance  of  existing 
interventions  or  policies,  (b)  asses  their  effects  and  impacts,  and  (c)  inform  decisions 
about  future  programming.  Evaluations  are  formal  analytical  endeavors  involving 
systematic  collection  and  analysis  of  qualitative  and  quantitative  information.3 

2.  The  systematic  and  objective  assessment  of  an  ongoing  or  completed  project,  program, 
or  policy;  its  design;  its  implementation;  and  results.  The  aim  is  to  determine  the 
relevance  and  fulfillment  of  objectives,  development  efficiency,  effectiveness,  impact, 
and  sustainability.  An  evaluation  should  provide  information  that  is  credible  and  useful, 
promoting  the  incorporation  of  lessons  learned  into  the  decisionmaking  process  of  both 
recipients  and  donors.  Also  refers  to  the  process  of  determining  the  worth  or  significance 
of  an  activity,  policy,  or  program.  May  involve  defining  appropriate  standards,  examining 
performance  against  those  standards,  assessing  actual  and  expected  results,  and 
identifying  relevant  lessons. b 

Related  terms:  assessment,  review 

The  transmission  of  evaluation  findings  to  parties  for  whom  it  is  relevant  and  useful  so 
as  to  facilitate  learning.  This  may  involve  the  collection  and  dissemination  of  findings, 
conclusions,  recommendations,  and  lessons  learned  from  experience.  In  the  context  of 
evaluation,  this  may  mean  sharing  the  evaluation  results  with  those  who  participated  in 
the  evaluation. c 

The  higher-order  objective  to  which  a  project,  program,  or  policy  is  intended  to 
contribute.3 

Related  term:  objective 

1.  A  result  or  effect  that  is  caused  by  or  attributable  to  a  project  or  program.  Often  used 
to  refer  to  the  higher-level  effects  of  a  program  in  the  medium  or  long  term  and  can  be 
intended  or  unintended,  positive  or  negative.3 

2.  The  overall  effect  of  achieving  specific  results.  In  some  situations,  it  includes  changes — 
planned  or  unplanned,  positive  or  negative,  direct  or  indirect,  primary  or  secondary — 
that  a  program  or  project  helped  to  bring  about.  In  others,  it  could  also  connote  the 
maintenance  of  a  current  condition,  assuming  that  the  condition  is  favorable.  Impact  is 
the  longer-term  or  ultimate  effect  attributable  to  a  program  or  project,  in  contrast  to 
outputs  or  expected  accomplishments,  which  are  shorter-term  effects. e 

Related  terms:  effect,  evaluation 

Specific  measures  that  assess  the  extent  to  which  a  project  accomplished  its  stated  goals 
and  objectives.  Also  called  impact,  outcome,  or  summative  evaluation.  Impact  evaluations 
focus  on  the  end  results  of  projects.  Questions  may  include  "What  is  the  result  of  the 
activities  undertaken  by  the  project?"  "What  happened  to  the  target  population  as  a 
result  of  those  activities?  Was  it  the  expected  outcome?"  "Should  different  activities  be 
substituted?"  To  measure  the  project's  effectiveness,  questions  may  include  "Was  the 
project  cost  effective?"  or  "What  would  have  happened  to  the  priority  group  in  the 
absence  of  the  project?"^ 

Note  that  impact  evaluation  is  frequently  based  on  an  experimental  design,  such  as  a 
randomized  control  trial,  to  more  confidently  infer  a  cause-effect  relationship  between 
program  interventions  and  observed  outcomes. 

1.  A  quantitative  or  qualitative  variable  that  provides  a  reliable  means  to  measure  a 
particular  phenomenon  or  attribute.3 

2.  Quantitative  or  qualitative  factor  or  variable  that  provides  a  simple,  reliable  basis  for 
assessing  achievement,  change,  or  performance.  A  unit  of  information  measured  over 
time  that  can  help  show  changes  in  a  specific  condition.  A  given  goal  or  objective  can 
have  multiple  indicators. c 

Related  term:  metric 


Indirect  effect 


The  unplanned  changes  brought  about  as  a  result  of  the  intervention.1 
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Term  Definition 


Inputs 

1.  Resources  provided  for  program  implementation.  Examples  include  money,  staff,  time, 
facilities,  equipment,  and  plans. c 

2.  The  financial,  human,  material,  technological,  and  information  resources  provided  by 
stakeholders  (i.e.,  donors,  program  implementers,  and  beneficiaries)  that  are  used  to 
implement  a  development  intervention.9 

Lessons  learned 

1.  Generalizations  based  on  an  evaluation  that  abstracts  project-specific  findings  to 
apply  to  broader  situations.  Frequently,  lessons  highlight  strengths  or  weaknesses 
in  preparation,  design,  and  implementation  that  affect  performance,  outcome,  and 
impact.3 

2.  Learning  from  experience  that  is  applicable  to  a  generic  situation  rather  than  a  specific 
circumstance.  The  identification  of  lessons  learned  relies  on  three  key  factors:  the 
accumulation  of  past  experiences  and  insights,  good  data  collection  instruments,  and 
context  analysis.9 

Related  term:  best  practice 

M&E 

See  "Monitoring  and  evaluation." 

Measure  of 
effectiveness  (MOE) 

A  criterion  used  to  assess  changes  in  system  behavior,  capability,  or  operational 
environment  that  is  tied  to  measuring  the  attainment  of  an  end  state,  achievement  of  an 
objective,  or  creation  of  an  effect. d 

Related  terms:  combat  assessment,  mission 

Measure  of 
performance  (MOP) 

A  criterion  used  to  assess  friendly  actions  tied  to  measuring  task  accomplishment.1-1 

Metric 

A  standard  of  measurement^ 

MOE 

See  "Measure  of  effectiveness." 

Monitoring 

1.  The  collection  and  analysis  of  routine  measurements  to  detect  changes  in  status.  It  is 
used  to  inform  managers  about  the  progress  of  an  ongoing  intervention  or  program  and 
to  detect  problems  that  may  be  addressed  through  corrective  action.3 

2.  The  regular  collection  and  analysis  of  information  to  assist  in  timely  decisionmaking, 
ensure  accountability,  and  provide  the  basis  for  evaluation  and  learning.  It  is  a  continuing 
function  that  uses  methodical  data  collection  to  provide  program  or  project  managers 
and  stakeholders  with  early  indications  of  progress  and  achievement  of  objectives. c 

Monitoring  and 
evaluation  (M&E) 

The  combination  of  monitoring  and  evaluation  that  provides  the  knowledge  required  for 
effective  project  management  and  reporting  and  accountability. c 

Monitoring  and 

An  overall  framework  to  ensure  that  M&E  outputs  make  a  valuable  contribution  to 

evaluation  (M&E)  plan  project  decisionmaking  and  learning.  The  plan  presents  the  necessary  supporting 


MOP 

conditions,  skills,  performance  and  learning  questions,  information-gathering 
requirements  (including  indicators),  resources,  and  activities  for  planning,  information 
gathering,  analysis,  synthesis,  and  reporting  processes. c 

See  "Measure  of  performance." 

Objective 

1.  The  clearly  defined,  decisive,  and  attainable  goal  toward  which  every  operation  is 
directed.1 

2.  The  specific  target  of  the  action  taken  (for  example,  terrain,  an  enemy  force,  or  an 
enemy  capability).1 

3.  A  specific  statement  detailing  the  desired  accomplishments  or  outcomes  of  a  project  at 
different  levels  (short  to  long  term).  A  good  objective  meets  the  criteria  of  being  impact- 
oriented,  measurable,  time-limited,  specific,  and  practical.  Objectives  can  be  arranged  in 
a  hierarchy.0 

Related  term:  target 
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Term 


Outcome 


Outputs 


Performance 


Performance  or 
process  indicator 


Performance 

measurement 


Performance 

monitoring 

Program 

Project 


Project  evaluation 


Project  or  program 
objective 


Definition 


1.  A  result  or  effect  that  is  caused  by  or  attributable  to  a  project,  program  or  policy.  Often 
used  to  refer  to  more  immediate  and  intended  effects.3 

2.  The  intended  or  achieved  short-  and  medium-term  effects  of  an  intervention's 
outputs,  usually  requiring  the  collective  effort  of  partners.  Outcomes  represent  changes 
in  development  conditions  that  occur  between  the  completion  of  outputs  and  the 
achievement  of  impact.9 

Related  terms:  effect,  impact,  output,  result 

1.  The  products,  goods,  and  services  that  result  from  an  intervention.9 

2.  A  final  product  or  service  delivered  by  a  program  or  project  to  beneficiaries,  such  as 
goods,  services,  training,  or  facilities  that  a  program  is  expected  to  produce  to  achieve  its 
expected  objectives.6 

Degree  to  which  a  development  intervention  or  development  partner  operates  according 
to  specific  criteria,  standards,  or  guidelines  or  achieves  results  in  accordance  with  stated 
goals  or  plans.13 

1.  A  particular  characteristic  or  dimension  used  to  measure  intended  changes. 

Performance  indicators  are  used  to  observe  progress  and  measure  actual  results  against 
expected  results.3 

2.  Specific  statistics  chosen  because  they  provide  valid,  practical,  and  comparable 
measures  of  progress  or  indicate  the  level  of  change  toward  achieving  expected  results  in 
a  given  period.  Used  to  measure  the  extent  to  which  goals  have  been  achieved.  Indicators 
correspond  to  the  expected  accomplishment  and  are  used  to  measure  performance.  One 
expected  accomplishment  can  have  multiple  indicators.6 

Related  terms:  achievement  indicator,  expected  accomplishment,  performance  measure 

1.  Ways  to  objectively  measure  a  program's  degree  of  success  in  achieving  its  stated 
objectives,  goals,  and  planned  program  activities.3 

2.  A  system  for  assessing  the  performance  of  development  interventions,  partnerships, 
or  policy  reforms  relative  to  planned  outputs  and  outcomes.  Performance  measurement 
relies  on  the  collection,  analysis,  interpretation,  and  reporting  of  data  on  performance 
indicators.9 

A  continuous  process  of  collecting  and  analyzing  data  to  compare  how  well  a  project, 
program,  or  policy  is  being  implemented  against  expected  results.13 

1.  A  set  of  interventions,  activities,  or  projects  that  are  typically  implemented  by  several 
parties  over  a  specified  period  and  may  cut  across  sectors,  themes,  or  geographic  areas.3 

2.  A  time-bound  intervention  similar  to  a  project  but  that  cuts  across  sectors,  themes,  or 
geographic  areas;  uses  a  multidisciplinary  approach;  involves  multiple  institutions;  and 
may  be  supported  by  several  different  funding  sources. 6 

1.  A  discrete  activity  (or  "development  intervention")  implemented  by  a  defined  set  of 
implementers  and  designed  to  achieve  specific  objectives  with  specified  resources  and 
implementation  schedules.  A  set  of  projects  make  up  the  portfolio  of  a  program.3 

2.  An  intervention  that  consists  of  a  set  of  planned,  interrelated  activities  designed  to 
achieve  defined  objectives  within  a  given  budget  and  specified  period.6 

Related  terms:  activity,  intervention 

Evaluation  of  an  individual  project  designed  to  achieve  specific  objectives  with  specified 
resources,  in  a  specified  time  span,  and  following  an  established  plan  of  action,  often 
within  the  framework  of  a  broader  program.6 

Related  term:  evaluation 

The  intended  physical,  financial,  institutional,  social,  environmental,  or  other 
development  results  to  which  a  project  or  program  is  expected  to  contribute.  Also 
referred  to  as  a  project  or  program's  "purpose."13 
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Term 

Definition 

Proxy  indicator 

1.  Indicators  that  show  indirectly  whether  a  result  has  been  achieved.  Used  when  it  is 
difficult  to  identify  direct  indicators  to  measure  the  result. e 

2.  A  variable  used  to  stand  in  for  one  that  is  difficult  to  measure  directly.9 

Result 

1.  The  intended  (or  unintended)  output,  outcome,  or  impact.3 

2.  The  measurable  output,  outcome,  or  impact  (intended  or  unintended,  positive  or 
negative)  of  a  development  intervention.*- 

Standard 

1.  Something  considered  by  an  authority  or  by  general  consent  as  a  basis  of  comparison.13 

2.  An  approved  model,  rule,  or  principle  that  is  used  as  a  basis  for  judgment. h 

Target 

1.  A  specified  result,  often  expressed  by  a  value  of  an  indicator,  that  a  project,  program, 
or  policy  is  intended  to  achieve.3 

2.  A  specified  objective  that  indicates  the  number,  timing,  and  location  of  what  is  to  be 
achieved.6 

NOTE:  Definitions  are  paraphrased. 
a  U.S.  Agency  for  International  Development,  2009. 
b  Organisation  for  Economic  Co-Operation  and  Development,  2002. 
c  International  Fund  for  Agricultural  Development,  2002. 
d  U.S.  Joint  Chiefs  of  Staff,  2008. 

e  United  Nations  Monitoring,  Evaluation  and  Consulting  Division,  2006. 

f  CORE  Group  Social  and  Behavioral  Change  Working  Group,  Glossary  of  Monitoring  and  Evaluation  Terms,  2008. 
9  United  Nations  Population  Fund,  2004. 

h  Webster's  Third  New  International  Dictionary  of  the  English  Language,  Unabridged,  Springfield,  Mass.,  1981. 

1  U.S.  Joint  Chiefs  of  Staff,  Joint  Operation  Planning,  Washington,  D.C.,  Joint  Publication  5-0,  December  26,  2006. 
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